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ABSTRACT 

A study assessed, the feasibility ox conducting a 
national cost-benefit analysis of vocational education and presented 
recommendations for— future research on the relationship between 
vocational education costs and benefits* Following an analysis of the 
state of the art in u tili zing cost-benefit methodologies to evaluate 
vocational education, a review, vas conducted of the. measurement , 0 
problems that might confront a study team performing a cost* benefit* 
analysis; then, after consultation vith technical experts, strategies 
for developing a cost-benefit model of vocational education vera 
formulated and i implemented. The model (which included variables in 
the 'areas of monetary benefits, non-pecuniary benefits, current 
costs', and opportunity— costs) ^then became the subject of -a Delphi 
analysis. Input vas collected from technical experts participating in 
the Delphi analysis vith respect to the desirability and feasibility 
of the components of the specified model as veil as the feasibility 
of overcoming some anticipated modeling and implementation problems. 
, Analysis of thse comments resulted in a series of recommendations for 
a national cost-benefit study.. The survey and its results are 
included in the study report. (Related project reports are available 
-separately through ERIC— see note;) (MN) 
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SECTION 1 
INTRODUCTION 



/OVERVIEW 



Cost-benefit analysis is one important method for improvi ng/esource 
allocation in the general .area of social welfare. The Department of Education 
has contracted with Rehab Group, Inc. for a study assessing *he feasibility of 
performing a national cost-benefit analysis of secondary, /postsecondary, ^nd 
Vdult vocational education. The components of this study include: 

• .An analysis of the measurement problems in performing a national 
cost-behefit study; \^ 

• An assessment of the state of the art in applying cost-benefit 
methodologies to vocational education; and 

• 'Recommendations concerning the feasibility of performing a national 
cost-benefit study. 

Each of these study components is examined in a separate document. The 
reports are written as companion pieces utilizing similar format and terminol- 
ogy. In addition, a final report synthesizes the major findings of all study^ 



areas into one document. 

FEASIBILITY REPORT OBJECTIVES 

Jhis paper has two primary objectives. One is to report the findings of 
.the Rehab study team on the feasibility of conducting a national cost-benefit 
analysis of vocational education. The other is to present recommendations 
for future research on the relationship between vocational education's costs 
and benefits. 

The conclusions drawn- are based on an interactive series of research 
tasks. The first relevant task was an extensive analysis of the state of the 
.art in utilizing cost-benefit methodologies to evaluate vocational education. 
The results of this state of the art overview are presented in a separate report 



ERJC 1-17 



9 

ERLC 



entitled Design of a National Cost-Benefit Stuc*y of Vocational Education at 
-the Secondary, Postsecondary and Adult Levels: State of the Art Report. 



< 



A second task, the results of which partially determine the conclusions 
y presented in this report, was a comprehensive review of the measurement 

problems that might confront a study team performing a national cost-benefit 
analysis of vocational education. The feasibility of conducting a national 
. study and the necessity for future research ace strongly influenced ,by the 
accuracy of available measures of the costs and benefits of vocational educa- 
tion. These measurement issues are reviewed in a companion report, Design 
of a National Cost-Benefit Study of Vocational Education at the Secondary, 
Postsecondary and Adult Levels: Cost-Benefit Measurement Report . 

A third source. contributing to the remarks made in this document was a 
Delphi study. This exercise solicited opinions from various technical experts 
in the areas of vocational educat.i.on_and/or cost-benefit analysis on the desi- 
rability and feasibility of numerous components of a cost-benefit model of voca- 
tional education. The Delphi methodology is'explained later in this report. 

FEASIBILITY REPORT FORMAT 

o ' , , " 

This report consists of four seet-ions-; — FtrHowing the introductory 
remarks, Section 2 discusses the steps in building a cost-benefit models of 
vocational education. The early steps in this process are then simulated 
to produce a preliminary specification of the potential components of a 
vocat^fhal education cost-benefit model . \^This specified model will be the 
subject of the Delphi analysis. The specification procedure draws technical 
guidance from the earlier work performed for the state of the art overview and 
the evaluation of cost-benefit measurement problems. Section 3 summarizes the 
opinions expressed by various technical experts in the Delphi exercise. 
These opinions focus on the desirability and feasibility of many of the 
components in the specified model, as well as on the feasibility of overcoming 
soma anticipated modeling and implementation problems. Section 4 presents 
* conclusions on the feasibility of. performing a national cost-benefit analysis 
of vocational education and recommendations for future research in the cost- 
" benefit area. 
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SECTION 



% BUILDING A COST-BENEFIjt MODEL OF VOCATIONAL EDUCATION 
INTRODUCTION * , 

\ 

In order to assess the feasibility of performing a national cost-benefit 
study^of vocational education, a preliminary investigation of the potential, 
components of the modej niust be undertaken. This is because the feasibility 
of ^conducting > national study depen ds upon the variables that make up the 
"model, the ability to operationalize these variables, and the availability of 
data to implement the model. 

A model specifies the variables that make up a functional system and 
the interrelationships between these variables. T&is section describes the 
model building process. First, it discusses the utility of .a. cost-benefit 
modef^and proposes a general modeling format. Second, it explains a strategy 
for developing a cost-benefit model that is consistent with that format. 
Third, it empjoys the modeling format and strategy to produce a very prelimi- 
nary, specification of a cost-benefit model of vocational education. The 
desirability and feasibility of this specification was analyzed using a 
Delphi analysis. The results of this analysis are reported in Section 3. 

UTILITY, OF <A COST- BENEFIT J10DEL 



The. utility of a well-designed model is multi-fold. First, it can illumi- 
nate, a comprehensive r^ange of policy options in that the model building pro- 
cess is an information generation. and problem identification process. A" 
policy decision that considers the issues raised in the modeling process may 
be made with more complete information than in other circumstances. Second, _ 
by specifying the particular factors in a functional system, the modeling pro- 
cess signals the type of technical expertise needed in the decisionmaking pro- 
cess. Consultation with the proper technical experts can contribute to a more 
educated policy decision. Third, since revenues are not infinite, policy 
makers must choose among alternative programs to allocate limited funds. It 
is quite rational to base such decisions, in part, on the relationship between 
program costs and benefits. Careful modeling can specify this relationship 
which then can be quantified using various cqst-benefit analytical techniques. 
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fourth, the presence of a model can defend. i\ decision maker against criticism. 
Policies are' of ten eyaluated based on the success of an outcome rather than the 
soundness of .a decision. Many sound decisions with the potential for positive 
outcome^y■B^fuce less than anticipated results due to intervening variables 
and stochastic events. Regardless of the outcome, few can argue with the 
wisdom of a decision based on weighing the expected advantages, and disadvan- 
tages before undertaking a course of action. 



-Co^t-.benafj.Lmod£.Ung__Cas^weJ \ .as. subseq uent cost-benefit analysis) is 



no.t a substitute 'for managerial judgment. Rather, it is a contributing factor 
to making sound management decisions. Cost-benefit modeling (and analysis') 
can help increase the information available to a .policy maker which results 
in superior decisions to those based solely oh subjective judgment. 



FORMAT OF A COST-BENEFIT MODEL 



This sub-section discusses the interrelationships among the components 
of a cost-benefit model. These components are: 



Theoretical sub-models 
Theoretically complete global model 
Operational global model 
Operational sub-models 



These interrelationships are displayed graphically in Figure 2.1. This figure 
-also illustrates the diversity of potential operational sub-models in any cost- 
benefit analysis. f 



The format proposed for constructing a cost-benefit model of vocational 
education ic influenced by the breadth of the vocational education enter- 
prise. Vocational education delivers services on secondary, postsecondary, 
and adult levels; offe/s over 400 course types in seven occupational program 
areas;-provides technical instruction in a variety of institutional settings; 
and teaches diverse student populations with.. varying educational needs. 
Because of this breadth, it is impossible to create, ong simple model to 
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Figure 2,1. Interrelationships Between the Theoretical Sub-Models, Theoretically Complete Global 
Model, Operational Global. Model, and Operational Sub-Models. 
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evaluate the costs and benefits of the entire realm of vocational education. 
Rathar, a series of theoretical sub-models with unique components must be 
designed. • * 

Wheirthe universe of theoretical sub-models is specified and logically 
interrelated, a theoretically complete global model exists. The theoretically 
complete global model reflects all the factors in the vocational education 
system regardless of the ability to measure or interpret them. It also 
characterizes the relationship between vocational education and the environ-' 
menr in which ,it operates. • 

It is- probable that some of the specified variables in a theoretically 
compl^e^alobaT model cannot be measured and/or some of the interrelationships 
cancr6t. be operational zed. This may be due to the- unavailability of data or 
'simply to> the lack of accurate measurement tools. In such instances, it is 
n6cessar?y to simplify, the model by creating an operational global model. This 
model includes ail factors of the functional system that can be measured and 
interpreted. Therefore, the operational global model trades off the thorough- 
ness of the theoretically complete global model in favor of practicality. It 
is the operational global model, rather th^n the theoretically complete global 
model, that' is the basis for executing a cost-benefit analysis. 

The operational global model is actually an aggregation of operational 
sub-models. Very often one or more of the sub-models is implemented in a 
cost-benefit analysis r-ather than the operational global model. Which of the 
sub-models are employed may depend upon what the particular research .question 
is, how the results will be utilized, and/or who is the potential user of the 
information resulting from the .analysis. 

» t 

Very often, the guidelines of a cost-benefit research project are so broad 
that they are almost global. Nevertheless, limited resources may force a study 
team working on such a projedt to choose among the various sub-models rather 
than implementing the operational global .opdel. In such cases, 'the universe of 
operational sub-models may be prioritized based on the- needs of the sponsoring 
agency, the desires of those in the field whb^wi^W use the results of the 
analysis, the opinions of technical experts,- or the logic of the study team. 



STRATEGY FOR MODEL DEVELOPMENT 

In order to maximize the effectiveness and validity of a model, the evalu- 
ation of a service, system utilizing cost-benefit modeling musf be based on a 
carefully specified, strategy for model development* One potential strategy 
is diagrammed in Figure 2.2. and discfussed subsequently . This strategy is 
based on the format of a theoretically complete global model, an operational 
global model, and their respective sub-models* 

* . ■ * 

Stage .One - Identify Model Requirements 

■ The first stage in model development is identifying the requirements for 
the model or model system. This necessitates delineation of 'the general pur- 
pose of the evaluative model, the potential users of the model, and the 
particular needs and concerns of the project team and potential user groups. 
As indicated, the model specification stage must be based 'on input from the 
potential users of the model rather than by the study team alone. This, will 
increase the chances that the final form' of the model will be responsive to 
the needs of its users. 

* 

Stage Two - Identify Anticipated Problems 

The second stage of model construction is the identification of antici- 
pated problems in. the design, operationalization, implementation, utilization, 
and evaluation of the model. Among the problems that are typically identified 
are the unavailability of. data, political constraints, disparity between the 
technical sophistication of the model builders and the model users, information 
processing limitations, financial restraints, reluctance of.ROtential users to 
accept the model, and inability to accurately measure all 'the costs and bene- 
fits of a program. Again, identifying potential problems shpuld.be a coopera- 
tive effort between the model builders and proposed model users. ,.Jf potential 
problems are, anticipated in advance, a study team can investigate alternatives 
that will maximize the validity of a model .given the projected restraints. 
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Figure 2.2>. The Cost-Benefit Model Development Process 
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Stages Three and Four - Specify and Evaluate Theoretical Sub-Models and 
Theoretically Complete Global Model 

Stage .three is thejpreliminary specification of a series^of theoretical 
•sub-models. This stage combines the conclusions about model requirements 
(stage one) and potential problems (stage two^) with technical information on 
the system being evaluated (e.g., vocational education) and the analytical 
approaches to relating program costs and benefits. After the preliminary 
specification, the models are reevaluated, refined, and adjusted. In stage 
four, the theoretical sub-models are integrated into a theoretically complete 
global model which is evaluated by the study team and potential users, and then 
further refined and adjusted. 4 

Stage Five - Assess Feasibility of Operationalizing Model 

i 

Once the theoretical sub-models and theoretically complete global model are 
.specified, the feasibility of creating an operational version of the model must 
be determined. This is done in stage five. It is appropriate for the poten- 
tial users, as well as the model builders, to have input into this decision. 

Stage Six - Identify Variables in Operational Model 

If construction of an operational model is deemed feasible, the next stage 
'in model development is to identify the variables to be incorporated into the 
operational model. Identification is £<fconplished by utilizing existing mea- 
sures of variables that have been employed successfully in past research or by 
generating new measures (which must then be tested for reliability and validity) 

> 

Stages Seven and Eight - Identify Interrelationships between Variables in ■ 
OperationalhModel and Create Hierarchy of Sub-Models 
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When variable identification is complete, the study team must construct 
the interrelationships between variables.. These interrelationships must be 
consistent with general theory in the system being studied and with statisti- 
cal theory. This stage culminates in the development of an operational global 
model. If only selected components of the operational global model are to be 
analyzed, a strategy for creating a hierarchy of sub-models must be developed. 

17 



When this is completed, the operational global model or the operational sub- 
models selected must be subjected to evaluation via simulation and field trial. 

Stage Nine - Simulations of Operational Model or Sub-Models 

It is recommended that two simulation steps be utilized. The first is an 
evaluation of the model using "perfect" .data fabricated specifically for this 
purpose. This artificial data set should be developed to reflect the range of 
possible model applications which might be found under real circumstances. 
This type of simulation will permit inspection of the model's ability to 
handle data arid withstand manipulation. After this simulation, the model 
should be reevaluated and necessary refinements made. 

. The second simulation should use "real" data, that is, information from an 
existing data set. At this stage, the behavior of the model in the context of 
imperfect data collected for other purposes can be observed. This may uncover 
unanticipated additional limitations of the model. This second simulation 
should be carried out through the analysis and interpretation phases so that a 
relatively complete judgment may be made concerning the internal and external 
validity and reliability of the model. At this time> the model should again 
be reevaluated and any necessary adjustments made. 

Stages Ten, Eleven, Twelve, Thirteen,, and Fourteen - Field Test Operational 
Model or Sub-Models, Identify Utilization Strategy, Implement Model (s), 
Evaluate Model (s), Make Recommendations I 

• ; \ 

The next stage in the model "development process is a field test under 
fully operational conditions. EfTis will provide a final examination of the 
quality of the model. The' field te^t should be implemented in diverse situa- 
tions which are representative of the -anticipated applications of the global 
operational model or operational sub-models. The results of the field test 
will be used to make final adjustments to the model priior to identifying a 
strategy for utilizing the model, implementing the model, evaluating the 
model, and submitting recommendations. 




THE BEGINNINGS OF A MODEL SPECIFICATION 



In order to help assess the feasibility of performing a national .cost-ben- 
efit analysis of vocational education, a very preliminary specification of the 
potential variables in this model was attempted* Once completed, the Delphi 
panel could evaluate the desirability and feasibility of each of the variables. 

To accomplish this preliminary specification, the study team simulated the 
first three stages in the model development process discussed Jn the previous 
sub-section. This simulation was hampered by the fact that the general purpose 
and potential users of the cost-benefit analysis (stage one) were not yet fully 
known. As a result, a series of hypothetical purposes and user groups were iden< 
tified to guide the model building process. These user groups along with their 
particular >needs are summarized .below: 

• The. Federal Government, whose needs might include allocating federal 
funds to the most efficient alternative programs. 

§ State governments* whose needs might include allocating state funds, 

and in the advent of block grants, federal funds, to the most effi- ' 

cient alternative programs. 
§ State education agencies, whose needs might include determining how 

to distribute school revenues^to^ maximize educational output in their 

schools. , 
§ Local education agencies, whose N needs might include making'efficient 

investments in alternative vocational programs. 

• Educational institutions, whose needs. might include increasing the 
efficiency of vocational BtagR'ams.^ 

t Individuals, whose j&ed/ might include determining whether vocational 
training will result in increased income, career advancement, or other 
benefits. 

§ Special needs populations, whose needs might include determining 
whether vocational training will result in various monetary and non- 
•pecuniary benefits. ^ ,„ < 

t 

Stage two of the development processcalls for the identification of anti- 
cipated problems in the design, op'erationalization, implementation, utilization 



and evaluation of the model* This problem identification is intended to be a 
joint task between the model builders and model users. Since th's is a simu- 
lation, and user groups are presently unknown, the study team substituted input 
from various technical experts in both vocational education and/or cost-benefit 
analysis. A long list of potential problems was identified by the study team 
and technical experts. The major problems are summarized subsequently: 

t Lack of available data types, particularly in the areas of program 
costs and student employment? and wage histories 

• Lack of follow-up data 

• Disparities in the quality and timeliness of data between states 

• Resistance in the field to use of VEDS data, which is the most 
recent attempt at national data reporting in vocational education 

• Lack of information on the duration and exposure of vocational 

4 

education 

• Lack of standard definitions of vocational education program 
enrollment 

• Difficulties* in developing a model that meets the neecjs of diverse 
user groups u 

• Problems in securing cooperation from potential user groups 

• Lack of acceptance or agreement among users of previous cost-benefit 
studies " „ 

• Measurement inconsistencies between alternative analytical approaches 
to relating costs and benefits " 

• Difficulties in selecting appropriate comparison groups 

• Difficulties in controlling for differences in non-educational 
variables between comparison groups 

• Difficulties in controlling for differences in program quality 

• Problems in treating the potential divergence between soci a| benefits 
and costs and private benefits and costs 

• Difficulties in measuring joint costs 

• Difficulties in choosing between average cost and marginal/ cost 

• methods <, / 

% Difficulties in calcul ating, the opportunity costs of vocational 
enrol Iment • x • / 

2-10 / 



t Difficulties in measuring the consumption benefits of vocational 
training 

• Difficulties in measuring non-pecuniary benefits and costs 

• Difficulties in translating non-pecuniary benefits and costs into 

f 

monetary values 

• Difficulties in interpreting the impact of an earnings multiplier 
effect 

• Problems in determining appropriate discount rates 

• Difficulties in form^ating a concise operational model given the 
breadth of vocational education 

• Financial restraints """" 

The last . two limitations suggest one additional problem. Since the vocational 
education enterprise is so diverse, a series of operational sub-models composed 
of different variables will need to-be developed. Given funding limitations* 
it is unlikely that all the sub-models can be implemented. Therefore, a final 
problem facing a national cost-benefit study teatf is determining an acceptable 
strategy to prioritize the sub-models. 

Stage three of the development process is the actual specification of the 
variables in the model and their interrelationships. In its most general form, 
a cost-benefit model of vocational education can be broken down into two func- 
tional equations: 

B = f (X^ ... X m , X n ... X z ) , where 

B = The benefits 'of vocational education 

^1 ••• X m ~ Monetary benefits 

*X n ... X z = Non-pecuniary benefits 



and 
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C - Ttie^ costs of vocational education 
¥].••• Ykk= Current costs ~ ^ 
Y n ... Y r ^Qapital costs 
Y s ... Yy = Opportunity costs 
Y z = Interest on school debt 




Table 2.1 breaks down each* of the broad categories included in the functional 
equations into its component parts. 

i 

Specifications of , the model also depend upon .the functional* relatloiTsJi'ips 
between variables. Among the factors that affect the nature of these func- 
tional relationships are: ^ 

t Selecting a measurement strategy for joint costs; 

• Selecting marginal or average cost methods; 
t Selecting an appropriate discount rate; 

• Choosing. comparison groups; 

• Choosing a unit of student participation; 

• Controlling for cost differentials between districts; 

t Controlling for differences in non-educational variables 

between students; • y 

- t Controlling for differences in program quality; — 

• Treating the divergence between social benefits and costs 
and private benefits and costs; and 

• Interpreting the impact of an earnings multiplier effect. 

Clearly, this specification of the model is preliminary and quite gen- 
eral. However, this broad specification is adequate to identify the basic 
components of a cost-benefit analysis of vocational education. The desir- 
ability and feasibility of utilizing these components in a national study 
can be assessed by soliciting reaction from a panel of experts -in the areas 
of vocational education and/or cost-benefit analysis. The results of such a 
survey of experts are reported in Section 3. * 
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TABLE 2.1. A Partial Listing of Potential Variables 
in a Cost-Benefit Model j)f Vocational Education 



Monetary Benefits 



Current Costs 



• Annual income 

• Fringe benefits (e.g., health 

insurance, vacations with pay) 

• Monetary benefits accruing to 
"students who enroll in vocational 
classes purely for consumption 
purposes 

Non-Pecuniary Benefits* 

• Greater job opportunities 

• Contentment with educational training 

• Higher job satisfaction 

• Positive work attitude 

• Employers 1 satisfaction with 

employee performance 

• Permanence of job 

• Lower crime rates 

• Better citizenship 

• Greater sense of well being- 



• Administration costs 

• Instructional costs 

• Costs of plant ^operation 

• Costs of plant maintenance 

• Fixed charges 

• Costs of other school services 

Capital Costs 
i 

• Building costs 

• Land acquisition costs 

• Costs for major equipment 

Opportunity Costs 

• Foregone Income 

• Cost of using plant for 

alternative purposes 
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SECTION 3 
DELPHI EXERCISE 



INTRODUCTION 



This section discusses the Delphi exercise that solicited informed 
opinions on ^desirability and feasibility .of various components in the 
cost- benefit model broadly specified in Section 2. The section opens with a 
general explanation of Delphi analysis. This i^followeci-by-a-descr-3ption-.of 
the survey methodology employed in this project. The section concludes with a 
summary of the results of the Delphi analysis. 

DELPHI ANALYSIS • s 

Delphi analysis is "a method for the- systematic solicitation and colla- 
tion of informed judgments on a particular topic." 1 In this methodology, 
information is usually collected from a respondent group through a survey- 
instrument. However A the methodology is significantly different from standard 
survey design, * \ „ 1" * . . • 

For^example, respondents are sent a series of questionnaires at estab- 
lished .intervals. -Each subsequent questionnaire builds on the issues raised 
or the responses received inrthe previous questionnaire. There are J:ypica]ly 
2 to 4 rounds 6f questions, although some Delphi exercises may be l'dnger. 

The Delphi methodology is also distinct from traditional survey designs >* 
because it includes a well-defined mechanism for group feedback: T^»at^ is, 
.respondents are usually sent a summary of the results of previous iterations 
of the questionnaire as well as any additional opinions volunteered by other 
panelists". The logifc behind the feedback component of the Delphi methodology 



t 



1 Murray Turoff, "the Design -of a Policy Delphi," Technological Forecasting 
and>Socia! Change . 2(1970), p. 149. . . " 
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is that it allows the diverse expertise of the" respondeat group to be shared ' 
with other panelists, and participants, to be informed of^he degree, of consensus ^ 
or polarisation * in' the group ♦ . . * 4 

A third unique aspect of the Delphi technique is that il encourages 
participants not -to feel constrained by the formal survey instrument* 
Respondents are encouraged to, critique, rewrite, or suggest new questions; to 
write justifications of their answers; and/or to include" general conranents on the" 
issues being discussed. _ — 

r 

f -> % , * * 

. The typical respondent (jroup in a Delphi exercise also varies from that , \ 

in a traditional survey* Respondents ar£* usual ly technical experts 'in a given . ; 

field or senipr mernbers of an organization. They- are selected specifically 

.because of their expertise^ and, therefore, are not a random sample, of the 

general .population. The numb'er of respondents in an av-erage Delphi analysis 

4 ranges between 10 and 50. ^ - 



The Delphi technique* may serve numerous objectives. Turoff suggests that 
these objectives include * ■ *; . * 

• Determining or developing a range of possible alternatives;* - ' 

• Exploring or exposing* underlying assumptions of infoVmation leading to 
differing judgments^; ; 

• a - Seeding out .information which may generate a judgmerited consensus, • * m 
on the part, of a respondent group; 

• ( Correlating informed .judgments on a topic spanning a wid'e range of - 

\% • * * « . *, 

disciplines; and'*'". •* . . « 

• Educating a respondent group as to the diverse and interrelated \ , , 
aspects. At. y topic. ; . I ^ ' ' ~- * 1 " 



2 Turoff, P. .149 
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The Delphi approach-has certain similarities to Ueci sion making by comit- 
^ tee* -In -both techniques a small group of experts attempts to reach a consensus 
. on. importanti v policy issues. However,, proponents of the Delphi method suggest 

ttfat it Kas certain 'advantages, over -decision by committee'.. For example, an out- 
^ ^spoken, personality c : arjnot dominate a Delphi exercise as*he/she can a committee 
meeting. In addition, respondents may be less hesitant /to. criticize opposing: 
"■views in a Delphvexercise since .anonymity is usually guaranteed. Similarly, 
sdoce respondents do hot meet face to face, ah individual may be less reluctant 
. to abaadon "6ne_jDOSitlon to support a second, based, on feedback of new evidence. 

/* 

* • <Use of the Delphi technique was pioneered in theearly l?60|s by researchers 
involved in technological forecasting*. The earliest'exercises asked respondents^ 
to predict when technological changes may take place. and the impact of the changes 
Sinceothat time, theielp'ht methodology has\become *an accepted 1 analytical tool; ■ 
in diverse* technological and policy areas.* " " 

* DESCRIPTION* OF THE^DELPHT METHODOLOGY . " 7 

„ " . " A Delphi .exercise was. not an original component of this stifdy's research 

* desi'gn- \The Delphi was .proposed in response to a change in study scope early 
in the project.. The Government Request for Proposal called for a project that 
would designed field test a cost^\benef it model applicable to a national study 

' of vocational education. Consistent with the modet development strategy dis- 
> % cussed earlier, the study Advisory Committee noted that the field testing of an 
operational model was premature until a -careful" analysis was made of the feasi- 
bility of building and implementing such a. model. Therefore, the study orienta- 
tion changed from field, testing a model to assessing the feasibility of a model. 
One tool proposed to assess this feasibility was a Delphi' analysis. 

* ** ", 

The Delphi methodology is not a substitute for careful analysis. Rather, 
it should be one component of a thorough analysis plan. Therefore, the Delphi 
exercise is just .one of several methods employed by this project to assess the - 
feasibility of performing a nation/1 cost-benefit analysis of vocational educa-' v 
tion. Other equally important factors are'an "evaluation of the state of,the 
art irf cost-benefit 'analysis and a rigorous review of potential measurement 
problems that was. based on an extensive.! iterature syrvey and informal convjer- ^ 
satio'ns with technical-experts. * ■ *• 



The fact that the Delphi analysis was -not an-original^part of the research 
design but a response to a change in study scope affected the size of the r.espo 
dent group. In order to maintain the established project schedule, the number 
of respondents was limited to nine, the maximum number allowable without under- 
going the tMiDe consuming process of obtaining Office of Management and Budget 
(0MB) approval of the survey instruments and design. The respondent group was 
made up of all. members of the projects Technical Advisory Committee, one membe 
of the agency sponsoring the research (the Office of Vocational and Adult Educa 
tion), and two representatives of state Departments of Vocational .Education. 
All members of the. Delphi panel were experts in vocational education and/or 
cost-benefit analysis. The names of the Delphi panelists are included 'in this 

report as* Appendix A-. • 

*} , ' 

The -Del phi exerciie ran for three rounds. The first two rounds consisted 
of a mail survey. Due to time limitations, the third round was scheduled as a 
conference at Rehab Corporate headquarters. . 

. Panelists were given approximately 9 days to respond to the mailed qtres^ 
tionnaires. Seven of the, nine panelists responded to the first round and all. 
panelists submitted round .two questionnaires. The Delphi design team utilized 
one week to feedback the results of round one to the respondents and to s^rt/c- 
ture and mail the second round questionnaire. An additional week was used to 
plan for the third round conference.. The entire Delphi process, from the first 
mailing to the third round conference, took just over six weeks. This does not 
include the considerable time spent planning and designing the exercise in ad- 
vance of the round one mailing. Prior to both mailings, all questions and 
instructions were pretested on co-workers of the Delphi design team. 



Jhesjnailing package for rounds one and two consisted of a cpyer letter, 
an explanation of the evaluation system used in the survey^and two copies 
of the questionnaire. The explanation of the evaluatiprr^system and the round 
one-and two questionnaires are exhibited in Appendix B. 

The. reasons for the duplicate questionnaire were threefold! First, it 
"couTd assis^a" panelist in planning his responses. Second, it could be used 
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as a record of a panelists responses' which later could be compared to those 
of the, overall group* Third,, after planning one's responses, the answers 
could be typed onto the second questionnaire in order to help assure the 
anonymity of respondents. 

• • ': 

The questions in all rounds were divided into three categories. These 
categories were: 

• The general design of a national cost-benefit study of vocational 
education 

r- Measurement issues and problems 

• Data availability v 

* 

However, the response mode differed for each round. In round one, panelists 
were instructed devaluate eacn response option to a question according to a 
desirability scale (very desirable, desirable, undesirable, and very upde- 
sirable) and a feasibility scale (definitely feasible, possibly feasible, 
possibly infeasible, definitely infeasib'le) . Note that no neutral answeV was 
available on the response scales. Each of the descriptors of desirability and 
feasibility was followed by a brief explanation or definition. The explana- 
tions are shown in Table 3.1. These explanations help establish comparability 
among responses even though the definitions may not be universally agreeable. 

/ In round two, panelists were asked to rank the desirability of each 
response pption in order of personal preference. No ties were permitted j 
between response options. Round two also included one series of open-ended 
questions. It allowed respondents to suggest important additional issues/ 
and questions in designing a national cost-benefit study of' vocational educa- 
tion that may have been overlooked by the design team.. These questions asked 
respondents to: 

♦ 

• List two. major obstacles in performing a national cost-benefit 
analysis of vocational education. - I 

• Describe a strategy for overcoming, minimizing, or dealing with each 
specified obstacle. ' j 

• Suggest two questions that should be addressed by a research team in 
designing a. national cost-benef it^analysis of vocational education. 



ERLC' 3.5 28 



/ 



1 



Table 3.1. An Explanation of the Evaluation 
System Used in the Delphi Questionnaires 



DESIRABILITY (EFFECTIVENESS OR BENEFITS) RESPONSE SCALE 
DESCRIPTOR . EXPLANATION 


Very Desirable 


will have a positive effect and little or 

no negative effect; 
extremely beneficial; 
justifiable on its own merit. 


Desirable 


will have a positive effect, negative 

effects are minor; 
beneficial ; 

justifiable as->a by-product or in conjunc- 
tion with other items. 


Undesirable 


will have a negative effect; 
harmful; 

may be justi-f/ied only as a by-product of 
a very desirable item, not justified as. 
a by-product of a desirable item. 


Very Undesirable 


will have a/major negative effect; 
extremely tyarmful; 
not justifjable. 


FEASIBILITY (PRACTICALITY/ RESPONSE SCALE 


DESCRIPTOR / 


' EXPLANATION 


Definitely Feasible 


no hindrance to implementation; 
no political roadblocks; 
acceptable to the public. 


Possibly Feasible 


some indication this- is implement able; 
^furttietj consideration or preparation 
* mustfbe given to political or public 
reaction. 


. Possibly Infeas.ible. 


some indication that this is Unworkable; 
siginificant unanswered questions. 


Definitely Infeasible u 


all indications are negative; 

unworkable;- 

cannot 'be implemented. 




~~ In the instructions accompanying the first and second round question- 
naires, respondents were encouraged to justify their responses; express 
opinions, rewrite questions, or suggest new questions* The instruction sheet 
explained that the questionnaire was "meant to be a stimulus for thought on 
the feasibility of performing a national cost-benefit analysis of vocational 
education." To facilitate and encourage comments, the questionnaire was laid 
out so that .the right hand page opposite each question was blank with room 
for commentary. The responses to and comments on all questions in rounds 
one and two are summarized in Appendix C. A transcript of\he round three 
conference has been 'submitted under separate cover. 

Many of the issues for the third round conference were developed from 
panelists 1 responses to the round two open-ended questions that asked for 
lists of -potential obstacles facing a national cost-benefit study team, 
strategies to overcome the obstacles, and additional questions that must be^ 
addressed in designing a national study. Responses to these third round 

questions .were solicited in the format of a round-table panel discussion. 

< 

Respondents were encouraged to express their views on each issue but were not 
required to participate in every aspect of the discussion. 

& The third round of the. Delphi exercise was plagued by the three major 
problems. First, the study team was reluctant to bring the panelists together 
for a face to face meeting. Although this was deemed an appropriate mechanism 
to summarize the issues debated in rounds one and two, it threatened the 
anonymity that had been established in the exercise. Nevertheless, the 
meeting was scheduled as a concession to project time constraints. Second, 
it was difficult to arrange a conference date that was amenable to all. nine 
panelists. Six of the nine participants committed themselves to attend the 
meeting on the date selected. Third, various last minute factors, including 

- the air traffic controllers strike, forced a number of committed respondents 
to miss the meeting. Anticipating the possible effects of the air traffic 
controllers strike, the study team decided to invite additional technical 
experts to the meeting. A total of six people attended the third round con- 
ference. However, only two of them had served as panelists for the earlier 
rounds. A list of conference attendees is included as Appendix D. The conference 
agenda is shown in Appendix E. 
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As- a result of the turnover in panelists, the conference functioned more 
as a fact finding meeting than as the third round of the Delphi exercise. 
Nevertheless, the meeting produced numerous contributions to assessing the 
feasibility of a national study. 

DELPHI RESULTS 

Section 2 specified several general characteristics of a cost-benefit 
model for a national study of vocational education. This sub-section reports 
the criticisms of a nine member Delphi panel on many of these characteristics. 

The Delphi analysis solicited responses from the panel of experts on the 
desirability and feasibility of several design, measurement, and data .base 
options. The panelists are all recognized experts in vocational education 
and/or cost-benefit, analysis. However, their opinions' must not be interpreted 
as necessarily representative of the vocational- education community at large. 
As is often. the case in a Delphi analysis, the size of the panel and the 
method of panel selection mitigates against the generalizibility of the 
results. Readers should, therefore, recognize the .limitations in these 
opinions. The limitations in the methodology were accepted a priori by the 
study team. This is why the Delphi survey was designed as but one of a series 
of components in assessing the feasibility of conducting a national study. 

Results on General Study Design 

As Section 2 indicated, potential users of a cost-benefit model should 
have input into its design at various stages of the development process. 
Therefore, identification of user groups will have a significant" impact on the 
ultimate specification of the model. The Delphi panelists ranked state 
agencies closely followed by the Federal Government as' the potential user 
groups most in need of the information that could be generated from a national 
cost-benefit model of vocational, education. It is apparent from some of the 
comments made by panelists that the current political arid economic environment 
influenced their rankings of potential user groups . Respondents who selected 
state and local governments cited their increased information needs based on 
the prospective growth in block grants. The choice of the Federal Government 
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was defended because of the need to make efficient budgetary decisions during 
a period of spending cuts* " 

A second. design issue examined in the Delphi is determining the optimal 
breadth of the proposed model. Nearly all respondents indicated through their 
comments that they are aware of the~ diversity of the vocational education 
enterprise. Given this diversity, the respondents were asked to choose 
between implementing...^ broad and versatile model that would provide meaningful 
results to many or all potential users and on varied .programs; a series of 
models that would separately address the information needs of different users 
and the characteristics of different programs; or a compact model that would 
focus on a single user, program area, program level, or delivery system. The 
panel favored the construction of several unique models. They also felt this 
type of model construction was the most/feasible alternative.. Interestingly, 
although the panelists rated a series of compact models first, they alter- 
natively preferred a broad. and versatile* model to a single compact model. 
Apparently, they feel that it is necessary to generate information on various^ 
elements of vocational education even if it means a trade off in the specificity 
of the model. 

It is important to realize that the issue raised in this question concerns 
determining the characteristics of the model to be implemented, not the 
-characteristics of -the model to be designed. According to-the strategy- for 
model development presented earlier, it is necessary to design a theoretically, 
complete global model. From that model, an operational global model and a 
series, of operational sub-models may be t constructed. A study team, in consulta- 
tion with potential user groups, may select which operational sub-models 
should be implemented. This model design process is summarized quite well by 
one of the Delphi respondents: 

A broad general model can be used, as a starting point for specifica- 
tions to meet particular needs and interests. Moreover, construction 
of'a narrowly focused model may be better achieved by specification of 
a general one (top down) than by ad hoc construction (bottom up). 
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•Three factors that could conceivably effect the breadth of the model 
design are the current availability of data, the level of available resources, 
and model construct capabil i ties ♦ The Delphi panel clearly concluded that in 
an ideal situation, cost considerations and current availability of data 
should be subordinate to model' construct capabilities in designing a cost- 
benefit study. However, data and funding limitations are a realistic concern^ 
One respondent's comments summarize these viewpoints: 



Given that 1) current data availability- and potential resources for 
the study pose severe programmatic constraints, and 2) the quatoty ..of- 
study activities and findings are dependent uppn a so1jd,^cp_mprehens^iY^_ 
model design, the consideration of model construct capabilities are^ 
paramount . Of course the delimiting factors cited^in .ppjnt l^data and 
resource availability) will necessitate flexibility^in the, development of 
the model . * * * - v " * - 

Should cost considerations dictate narrowing, tHe scope of the study to 
one particular education level, .the, ; pan;e;l:tst's favored examining secondary 
vocational education first, "post s^ and 
adult vocational education a distant third. The Delphi Respondents' clearly 
rejected the option of an r: gggregated.^OTjnati.pn r of secondary, postsecondary , 
and adult vocational program?lis" ? "a cost-savang alternative. .In the^words : ,of 
one panelist: * * r - J. ^ 



The types of benefits differ considerably -by institutional level": 'For 
example, while job pi acement rates jtnd^eanning levels misht be the most^" 
appropriate benefit measures .for postsecondary and adulXf^6"grams : , /the 
benefit of secondary programs mfght be ,mos^b appropr.iately ^dgetf^by • 
levels of skill proficiency or attitudinal^ch^qges/ Consequently/ '1 do 
not see how an aggregate benefit assessment across^iristitut'lonal levels 
* could be fairly constructed^ \ : " ~ ~ - 

Under ideal conditions, respondents feel that the mosf "informative study . 
should include and distinguish between the various program levels of vocational 
education. They similarly believe that the potentially^ differing efficiencies 
among vocational program types and delivery systems should be analyzed as part 
of a national cost-benefit study. Concerning vocational program types,, 
respondents indicated a desire to distinguish between the returns of specific 
programs within broad'program areas. However, they assessed this distinction 

r 

"as potentially unworkable and, therefore, infeasible. Thus, distinctions 
between programs may have to be made between broad program areas only. 
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The model specified in Sectiorr 2 broke vocational education benefits into 
two categories: economic benefits, which can be measured by annual income, 
and non-pecuniary ^enefits. B^sed on various^ respondent comments, non- 
pecuniary benefits appear to be the most difficult aspect of the study design 
to handle. Clearly, panelists feel that they should be a component of the 
study. Interestingly, although there is great concern over how to incorporate 
non-pecuniary benefits into the analysis, and strong criticisms .aimed at cost- 
benefit analysis for its inability to reflect these benefits, the consensus of 
the panel is that such incorporation in some form is feasible. Apparently, 
this viewpoint is based on the increased attempts to operationalize non- 
pecuniary costs and benefits in existing cost-benefit studies. As one 
panelist comments, "Multi-criterion benefit-cost models are beginning to 
emerge and should be looked into. 11 

A second .issue raised .in the model specification was how to treat joint 
costs. Joint costs are costs incurred when an educational input, such as a 
piece of equipment or school building, is used by more than one student 
group. Allocation of the joint costs presents a difficult measurement problem. 
Several "treatment options exist, including excluding t^an from analysis, 
evaluating the marginal cost of their use, evaluating the average cost of 
their use, and evaluating them using game theory. Average cost of use was 
the most desirable method of evaluating joint costs to the panelists, with 
marginal cost of use a close second. Since marginal and average cost;methods 
may be relevant in different situations, an optimal alternative might be to 
use both costing techniques. One panelist, using similar logic, called for 
the judicious use of average costing, marginal costing, and game, theory in a 
cost benefit analysis: 

* For starting a new added program, marginal costs may be the best; for 
evaluating a whole system, average cost is attractive; game theory 
methods are relevant when considering several different added programs or 
combinations thereof. 

A" third component of the model specified in the previous section was a 
discount rate. Utilizing a.discount rate in cost-benefit analysis permits 
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the evaluator to equate future income with present values. The panelists 
favored using the rate of inflation as the means of measuring the discount 
rate. This option was more desirable than either the prime rate of interest 
or the rate of interest on government treasury bills. Surprisingly, the 
overalTsecond choice of the Delphi panelists was to exclude a discount rate 
from the study. However, there was extreme polarization on this response 
category. 

As specified in the model, a student may be enrolled in vocational 
education both for investment and consumption reasons. Some critics have 
contended thatjt is unreasonable to support expensive vocational programs on 
the basis of non-investment benefits when non-vocational education programs 
are being underfunded. Panelists, however, supported the presence of consump- 
tion ^benefits in the model. , Nevertheless, they rated the feasibility of 
accurately measuring the level of consumption benefits as quite low. 

A vocational education graduate's increased earnings will have a ripple 
effect throughout the economy, as he/she spends money and increases someone 
else's income. This was termed an earnings multiplier effect in the model 
specification. The panelists judged that this earnings multiplier effect 
should be considered in a cost-benefit -study. They did recognize, however, 
that consideration, while desirable, is somewhat less feasible because of its 
measurement difficulty. 

The opportunity cost of attending a vocational education program may 
er-i-.er into a cost-benefit model as one of the largest cosT~components. "The 
panelists concurred that use of foregone income as a measure of the opportunity 
costs of. attending school was desirable arid relatively feasible. The proxy 
for foregone i ncome* deemed most desirable was the average earnings 'of individ- 
uals with similar characteristics who "are not attending school. 

The model specification sub-section also suggested that social costs and 
benefits may diverge from private costs and benefits. Therefore, determination 
of which entity is the proper basis for a cost-benefit analysis will impact 
the study results. Panelists indicated that measurement of both -private and 
social costs and benefits are desirable and feasible in a national study. 
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Another model specification issue t^at will have serious implications on 
the succeeding analysis is the choice' of a comparison group. Concerning 
secondary vocational education, respondents concluded that the most" logical 
comparison group was students in a general education program. However, 
the panelists were somewhat tefoperate in their support of this option in that 
.comparisons with other alternatives were ranked just below general education 
programs . These included students attending a college preparatory program, 
individuals not attending secondary school, and a weighted average of all 
three activities. For all comparison formats, panelists raised definitional 
and data availability problems in measurement. 

Panelists were evenly divided between students in two year general 
curriculum colleges and individuals not attending postsecondary schools in 
their choice of an optimal comparison group for postsecondary vocational 
education. Regardless of the level of education analyzed or the choice of 
comparison' group, it is important to attempt to control for differences on 
non-educational variables between groups. 

In an effort to distinguish between program enrollees and individual 
course takers, respondents supported "enrollment in a fixed series of related 
vocational classes" as a superior definition of a program participant for ^he 
model. Further, they agreed" that full time equivalent (FTE) students was a more* 
suitable method for counting students than either ADA,,J\IM r or the average of 
ADA and ADM. An alternative measurement format, was^ suggested by one panelist: 

FTE is an excellent measure of load orrth'e System. However, seriousness 
of participants is measured by average daity attendance. I suggest (as 
an alternative) the measure: 

(Number of hours per week) -X^ Number of enrollees) X R 

where R is a reduction factor to account for absentees; R should .probably 
" not be linear. " , ^, 

• One final factor examined In the Delphi that could impact .on the results 
of a cost-benefit study is* the treatment (and possible w£ighting)~of 'diff ereflces 
in program quality. Measuring differences in the "quatjty. of< vocational . 



programs was judged to be highly desirable yet possibly infeasible by the 
majority of panelists* Panelists emphasized the^need for delicacy in program 
quality measurement criteria, noting the potential political impact of "such 
measures. . » 

1 Results on Data Availability ^ 

There are several sources of data that could be used in a national - 
cost-benefit study of vocational education. Delphi panelists specified that 
utilizing existing data bases -supplemented by some new data collection was the 
preferable strategy for securing data in a national study, fhis option was 
preferred to relying solely on-existi,ng data bases or conducting, a data 
collection survey exclusively for the national study. . 
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Respondents were also queried on the desirability and feasibility of 
usinga number of different existing sources as the basis for the national 
study'-s data. These sources were; 

• National Center for Education Statistics' (NCES) Vocational Education 
Data System" -(VEDS) ' . 

• Bureau cf Occupational and Adult Education's (BOAE) Statistical 
Reports (1973-1978.) 

• NCES' High School and Beyond Longitudinal Survey (1980) 

• Department of Labor's (DOL) National Longitudinal Survey (1979) 

• NCES'' National Longitudinal Survey of the High-School Class of 
1972 

• National Institute of Education's (NIE) Survey of Vocational Schools 
in Ten States (1980) 

• NCES 1 Survey of Non-collegiate Pcstsecondary Students and Schools 

(1972-1980) , 
•. Assistant Secretary for Planning and Evaluation's (ASPE) Survey of 
Vocational iEducation Students and Teachers (1972) 

• Office of Civil Ri.ghts' (OCR) Survey of Vocational Education Schools 
(1979) 



• Office of Education's (0E) "437 Files" (Grants and Expenditures under 
• State Administered Programs) 
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• Census Bureau's Current Population Survey Supplement 
- • Project- Talent Data Base* 

• ~*NCES' Survey .of Course Offerings and Enrollments .(1973) 

• Survey Research Center's Youth in Transition Data Base (1966) 

No sound conclusions were made by the Delphi panel about the desirability 
or feasibility of using these various sources* Rather, many respondents 
expressed uncertainty about the contents of the alternative data bases. . 
It is interesting to note, however, that of the four respondents knowledgable : \ 
about VEDS, two ra*od the data source undesirable. VEDS has come under sharp 
attack by many in the field for being duplicative and unnecessary. 
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• SECTION 4- p 

RECOMMENDATIONS ■ ~ ' v '* ' 

' ■ _ - 

FEASIBILITY OF PERFORMING A NATIONAL STUDY : \ • 

t 

The results of the state of the art review, assessment ^of potential 
measurement problems*;, and Delphi analysis suggest that a national cost^benefdt 
study of vocational' education is technically feasible. However, this assess- 1 
ment must be viewed in terms of the current level of sophistication i.n 
relating costs and benefit's. - , 

Cost-benefit analysis, based on existing technologies, is an imperfect 
analytical tool* Not all theoretically appropriate .variables in a cost- 
benefit model may be operational i zed. Other variables mayjbe operational ized 
bit only by using imprecise proxy variables. Consequently, in most cost- 
Benefit analyses dealing with social issues, there is a significant deviation 
between the theoretically complete global model and the. operational global s 
Jmodel. Nevertheless, most modeling, measurement, and data obstacles can , be 
/overcome- to the point where the product of a cost-benefit analysts is useful 
and reliable. ^ ^ 

One of the companion reports in this study, Design of a National Cost- 
Benefit Study of Vocational Education "atTthe Secondary, JlostTecondary, and 
Adult Levels: State of the Art Report , illustrates that a large number of 
cost-benefit analyses of vocational education i ave been conducted on a sub- 
nationaf level. Although the logistics of a national study will* be substan- ' 
tially more imposing than those on a sub-national *basis, they both must 
cpnfront many of the same technical obstacles. The; smaller-studies have" 
provehHhat these limitations can be surmounted. They also demonstrate . 
that a cost-benefit study can contribute to the. understanding of vocational 
education. 

A national study faces numerous unique difficulties as- well. Many of 
these problems are addressed in the last sub-section on the recommendations 
for a national cost-benefit study. These problems must be given careful 



Attention by .a. national cost-benefit study team. Again, however, they are not 
fatal to executing *tf national study! 

* ' * * •» 

UTILITY OF PERFORMING A 'NATIONAL STUDY • 

A national study of the costs and benefits of vocational education 
should not be Implemented solely based on its technical feasibility. In' 
addition, the utility of a national study must be assessed prior to committing 

scarce revenues to the research. The utility, of performing a. national study 

. > ^ «" ^ • - 

is 'considered in* this sub-section. »*■ • 

A national cost-benefit study should plpgve useful for a number of reasons. 
For exajiple, cost-benefit' analysis of vocational education can contribute to 
sounder policy decisions. --The results of a cost-benefit analysis, even if 
based on an imperfect model, .can lead to superior dec.isio.ns than those -b'aseH 

merely on subject'ive judgment. ^ 

' * ' \ • 

J i > - . ' . . , 

Second, the results can be used on the sub-national' as well -as national 

level. With the prospect of increased block grants in education, state and ' 

local agencies need more information on the* relationship between sprograip' costs 

and benefits in order to help make, good ; policy decisions. Given scarce ; / 

resources, the relationship between costs 4 and benefits is a rational basis' on 

which to make such decisions. Therefore, *a national study can. contribute to 

more informed decision-making at the -state; ana. J oqM levels; t . • t 

* « • • . . . 

Third, in the process of building a; theoretical model of the costs an^ 
benefits of vocational education, a study team can help pinpoint crucial data 
needs. That is, in .an effort to operationalize'the model, the study team, must 
assess which data are available, which are reliable, and which are duplicative. 
This identification process can potentially contribute to reducing the data 
burden that currently exists in vocational education.' , # , 

Fourth, the results of a nationaT cost-benefit study will complement * 
existing, evaluative research oh vocational education*. In par,ticul ar,^the 
national study wiil be a useful adjunct to the research conducted by the 
Congressional ly mandated NIE Vocational Education Stirdy: The information .. 



generated- by the two, studies wiVT produce a wealth of data on the present 
state of vocational* education, 

•As with any. analytical technique, there is the possibility that the 
results of a cost-benefit analysis can be misused. For example, some may 
treat the results of such .an analysis as a magic formula that can conclusively 
allocate scarce funds among alternative programs. The methodological limita- 
tions inherent in the technique are too g^eat to base such decisions solely on 
the results of a cost-benefit analysis. Nevertheless, cost-benefit analysis 
can provide significant input into making such policy decisions. That is, 
when .used as one component in a mul ti-criterja policy evaluation, rather than 
indiscriminately, cost-benefit analysis can be an informative poTi cy-relevant 
tool. % 

i e 

RECOMMENDATIONS FOR A NATIONAL COST-BENEFIT STUDY 

This sub-section presents recommendations for consideration in planning a 
national costrbenefit study of vocational education. These recommendations 
pertain to funding limitations, user groups, data problems, modeling^ considera- 
tions, and measurement problems. 

.Funding Limitations* - 

As a result of Federal efforts to balance the budget, revenues for program 
evaluation are becoming scarce. Ironically, program evaluation methodologies . 
can help policy makers allocate scarce dollars more intelligently. Given the 
current fiscal environment', implementing a global operational model, although 
the optimal choice in designing a' nationaKcost-benef it analysis of vocational 
education, is improbable... Th^Vefore, a hierarchy of sub-models must be 
created and the sub-models implemented subject to available funding. Given 
present funding limitations, the following recommendations are made: 

* • The cost-benefit study should not be conducted on a national basis, 
*but rather, with national considerations. Therefore, a sampling plan 
must be developed that represents the many diverse characteristics of 
the vocational education enterprise. 



t If a choice must be made among progran/ levels, the first priority 

should be an analysis of secondary'vocational education* This is d 
because secondary vocational education has a' higher enrollment, 
utilizes more revenues,, and probably has more thorough and accessible 1 
data than postsecondary or adult vocational education. 
. • Since a national study -will be Federally funded, the analysis must- 
first serve Federal policy determination needs. It is preferable, 
however, for, the study to meet the needs of more diverse users. 

• If a national study cannot be /funded, *.*a less costly alternative might 
be to provide technical assistance to the states to help them develop 
the skills to conduct thei.r^own cost-benefit analyses. With the 
prospective advent of block grants, this investment in capacity- 
building 'on the state level should prove beneficial. 

User Groups 

^Information from a national cost-benefit study can be used by diverse 
groups, iniluding the Federal Governmen|, state agencies, Ideal agencies, 
--p^raTEsTrrfd students, and special needs populations. The following, recommen- 
dations are made concerning user groups: 

• If funding levels parmit, at a minimum the study, should be designed to 
fulfill the needs of both the Federal Government and state agencies. 
The information needs of state agencies will be increasing given the 
current fiscal and political environment. 

• Given existing set-aside requirements for special populations, the study 
should attempt to serve the needs of state and Federal special education 
program administrators in the area of vocational education. 

• In order to increase the acceptance of the study by potential users, 
user groups should have substantial input into the design 'of the 
cost-benefit model. 

Data Problems 

• 

Disparities in the availability and quality of data among states is a 
serious obstacle to perf ormi ng a national cost-benefit study. The following 
• recommendations address these and various other data problems: 

' 42 
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• Since it has been proposed that a national study be conducted on a 
sample basis, the sampling plan should reflect an awareness of data 
availability and data quality disparities. If possible, states should 
be included in the sample only if they have available: 

- data on program costs 

- data on student employment and wage histories 

- reliable enrollment data 

- enrollment data sensitive to differences in duration and 
- exposure 

- student follow-up data 

- data files that are updated regularly 

• Accurate definitions of varipus data types must be created. The 
study team must be sensitive to possible inconsistencies in defini- 
tions between states. 

• The study should utilize existing data wherever possible and only sup- 
plement these data with new data collection if necessary. New data 
collection should be kept to a minimum given the current attitude at 
the state and local level that tqo much duplicative and unimportant 
.data are already demanded. 

• One task in the national study can be identifying unreliable and 
duplicative data elements that are collected through national 
reporting mechanisms, particularly VEDS which is the newest and 
perhaps most criticized mechanism. This information could be gathered 
as a byproduct of scrutinizing national data sources for possible use 
in the cost-benefit analysis. 

• A parallel study should be funded that utilizes the results of the 
previous task and formulates strategies to reduce the data reporting 
burden faced by states and localities. These strategies might include:, 

- creating a vocational education management information system 
(MIS) to process available data more efficiently 

- -adding or deleting data types in statutory reporting systems 

- Standardizing acceptable surrogates of unavailable or unreliable 
data * 

- standardizing data definitions 

- standardizing data reporting requirements 
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Cooperation with the cost-benefit study team and, therefore, accept- 
ance .of the study findings may increase if user groups are shown that 
the research will help reduce their data reporting burden. 

Modeling Considerations 

For cost-benefit research to be most valuable, a strong commitment must 
be made to a thorough model development process. Very often, knowledge gained 
from the modeling process is as significant as the actual results of a cost- 
benefit study. The following recommendations are made pertaining to model 
development: . . 

• To maximize the acceptance of the model, the model building process 
must be fully documented. 

• A national cost-benefit study should be required to include specifi- 
cation of a theoretically complete global model, creation of an 
operational global model and sub-models, moclel simulation, model field 
testing, implementation, and evaluation. 

To best execute this comprehensive process, it may be preferable to fund a 
series of consecutive studies, each performing one or more steps in the 
modeling process, rather than one major study. 

Measurement Problems 

There are numerous obstacles to measuring accurately the costs and 
benefits of vocational education in a national cost-benefit study. However, 
many of these obstacles can be overcome, or their effects acceptably minimized 
by a knowledgable model building team. A number of recommendations are made 
concerning measurement issues: 

• Multiple analytical methods should be employed to assess the relation- 
ship between vocational education costs and benefits. This is because 
alternative analytical' approaches can produce varying results under 
certain conditions. 
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• The study must include both monetary and non-pecuniary costs and 
benefits in its design. Although measurement of the latter is diffi- 
cult, there are numerous acceptable proxy variables that represent 

* 

non-pecuniary costs and. benefits. A model that dismisses non-pecuniary 
costs and benefits with the disclaimer that "since they can't be 
measured, they will be omitted" is seriously deficient. 

• In a cost-benefit model of vocational education, both private and 
social costs and benefits must be calculated. 

• Since each may be appropriate in different situations, bot.i average 
cost and marginal cost methods should be included in the analysis. 

• The cost-benefit study team should attempt to incorporate game theory 
in allocating joint costs. The assumption that allocating joint costs 
is not a problem for the study since the marginal costs are zero, is not 
appropriate under all conditions, such as in allocating costs among 



• Because of the breadth ot^vjacational education, overaggregated data 
collection and analysis must be avoided. A cost-benefit analysis must 
recognize the potentially varying efficiencies of vocational education 
by program level, program area, delivery system, and student population 
group. 

• It is feasible to include in the cost-benefit model a rate that dis- 
counts future benefits to present values and the opportunity costs of 
participating in vocational programs (measured, fo\^xample, by 
foregone income). ^ 

• Model builders should investigate the possibility of including in 
the model consumption benefits, an earnings multiplier effect, and a 
control for differences in the quality of vocational programs. Inclu- 
ding these factors may not be feasible. However, their omission from 
the model is not fatal . 



ongoing programs. 
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DELPHI PANEL MEMBERS 



Dr. Kern Alexander 

Director " * * 
Institute for Educational Finance 
University of Florida ' 
Gainesville, Florida ' 

Mr. Don K. Gentry*. 

State Director of Vocational Education 
Ind(^flapol is, Indiana 

Dr. George Hagerty 

Advocate for- Vocational Career Education 
Division of Personnel- Preparation 
U.S. Department of Education 
Washington, D.C. 

Dr. Xharles Hopkins 

Oklahoma Department of Vocational Education 
Stillwater, Oklahoma 

Dr. Jin Eun Kim 

Assistant professor of Educational Administration 

School of Education 

Catholic University of America 

Washington, D.C. 

Dr. Gary Meers 

Director, Special Vocational Needs 

The. Center for Vocational Teacher Education. - 

University of Nebraska 

Lincoln, Nebraska 

Dr. L". Allen .Phelps 
Assistant Professor 

Department of Vocational and Technical Education 
University of Illinois 
Urbana-Champaign, Illinois 

Dr. Robert Thrall 
Professor and Chairman 
Department of Mathematical Sciences 
Rice University 
Houston, Texas 
i 
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APPENDIX B 
DELPHI SURVEY PACKAGE 



Explanation of the Evaluation System for Round One 
Round One Questionnaire 
Round Two Questionnaire • 



EXPLANATION OF -EVALUATION SYSTEM 



The following evaluation system will be used throughout the exercise to 
provide possible expressions of judgment* Please keep the following 
guidelines in mind when responding to each question or using the des- 
criptors in a comment. This is important in establishing comparability 
among responses even though the definitions may not be universal ly agreeabl 



DESIRABILITY (Effectiveness or benefits) 



4 


vcijr ucj i{ auic 


will have a no^itive effect and little or no 

Infill 1 IU V U U LsUJIblVw >* 1 1 >* \« l# VAIIVJ 1 1 l# l# 1 s* w 1 llw 

negative effect; 
extremely beneficial; 
justifiable on its own merit. 


3 


Desirable 


will have a positive effect, negatjve effects 

are minor; 
beneficial; 

justifiable as a by-product or in conjunction 
with other items. 


2 


Undesirable 


will.havea negative effect; 
harmful ; ~ . 

may be justified only as a by-product of a 
very desirable item, not, justified as a 
by-product of a desirable item; fc " 


1 


Very Undesirable 


will have a major negative effect; 
extremely harmful ; 
not justifiable. 




FEASIBILITY (Practicality) 




4 


Definitely Feasible 


no hindrance^ to implementation; 
no political roadblocks; 
acceptable to the public. 


3 


Possibly Feasible 


some indication this is implementable; 
further consideration or preparation must 
be given to political or public reaction* 


2 


Possibly Infeasible 


some indication that this is unworkable; 
significant' unanswered questions. 


1 


Definitely Infeasible 


all indications are negative; 
• unworkable; 

cannot be implemented. 
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DELPHI QUESTIONNAIRE . 

ROUND i • • 

for the 

Design of a National Cost-8enefit 
Study of Vocational Education 
at the Secondary, Postsecondary, 
' and Adult Levels 



Rehab Group, Inc. - 
5827 Columbia Pike 
Falls Church, Virginia 22041 




-er|c 



June 26, 1981 
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INSTRUCTIONS FOR OELPHI PANELISTS 



(1) Enclosed are two copies of v the Oelphi questionnaire. Return only one in 
the enclosed stamped and preaddressed envelope. The second is for refer- 
ence and .to assist you in preparing . your response. * 

(2) Please mail your response on or before July 7 . 

(3) The questionnaire is divided into three sections: 

t Questions on the general .design of a cost-benefit study of 
vocational' education 

t Questions concerning measurement issues and problems 
• Questions on data availability 

For each question, you are to evaluate the desirability arid/or feasibility 
of every response according to an evaluation system. This evaluation system 
is explained on a separate- sheet so that you can refer to it easily 
throughout the exercise. Be sure that '^ou evaluate every response; do 
not merely select vthe one response that is most agreeable to you. 




(5) 



(6) 



You are encouraged to write justifications for your answers and general 
comments on the issues discussed in each question. Such comments are an 
important part of the information collection process. The amount of 
information gained from the Oelphi exercise is dependent upon each res- * 
pondent writing, relevant comments on the questions. These comments will 
be made available to. other panelists before they respond to the second 
round questionnaire. Anonymity will be maintained in a 1 1 cases. The 
right hand page opposite each 'question is blank so that you can easily 
write your comments. Feel free to attach additional sheets, if necessary. 



(7) 



Questions in this Oelphi exercise are, meant to be a stimulus for thought 
on the feasibility of performing a national Gost-benefit analysis of- 
vocational education. You. shoold not feel constrained by the questions. 
In fact, you have the following options on any question: 
\ < w 

• Rewrite the question and answer your version if you feel 

the v original is misleading or inappropriate. * 

t Suggest questions you would like to see. in the next roupd 4 
of theXexercise -that you feel will .clarify an tissue or * 
raise a\new alternative that the Oelphi panelists should 
consider^ 

# Write comments that relate to the question or that clarify 
your response to the- question. 

In subsequent rounds, additional questions will be developed that attempt 
to highlight reasons-why polarization of viewpoints occurred on some 
issues. Also, new questions will be added or old questions rewoKded to 
clarify viewpoints. Therefore, the Delphi process is a cumulative one. 

Thank you again for your commitment to this exercise. We look forward 
to your response. If you have any questions, feel free to call Dr. 
Mark Shugoll or Mr. Tim Helms, collect at (703) 820-4350. 
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QUESTIONS ON GENERAL STUDY DESIGN 

The following questions deal with, general issues in the design of a 

national cost-benefit study of vocational education, 

i 

r. - A- national cost-benefit study of vocational education must be designed 
to meet the needs of its users. Please evaluate the desirability and 
feasibility of designing a study which would yield information to meet 
the needs of the following user groups: 

Feasibility 



-' Individuals , whose needs might include 
determining whether vocational training* 
.will result iri increased future benefits 

Educational institutions , whose needs 
might include making efficient investment 
decisions 

- Local education agencies , whose needs 
might include making program decisions 
-based on J deal manpower needs. 

- State education agencies , whose needs 
might "include determining how to 
distribute* educational revenues to 
maximize educational output 

- Federal .Government , whose needs might 
include allocating scarce resources 
among alternative programs 

- Other (please specify) 


















» 
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Please evaluate the desirability and feasibility of each of the # 
following possibilities in designing a national cost-benefit study 

of -.vocational education: „ 

\\ , Desirability Feasibility 

- Narrow the focus, of the; study to a single \ ■* 

us4 : r and construct a compact" models ^_ 1 — 

- Develop a broad' and versatile model 
that would proVicJe results that are 
meaningful .to jnaijy or all 'potential 

users. ' \ : r 

- Construct several models that sepa- , - / 
natel&addr'essi the information needs 

of different users ' * * _^ _J , - 



'1 . * ■ 
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Please .evaluate the desirabilitV 'of each of the following considerations 
in designing a national cost-benefit study of vocational educati 

' Desirability . 

- Study design. should* be dictated by c the , 
current availability of data _ * 

- Study design shpuld be* dictated ;by model,^ 
construct capabilities ^ — . 

- Study design' should be dictated by 
cost considerations 
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Trie scope of a national cost-benefit evaluation is of particular 
concern* The larger the scope, the more generalizable are the 
results. However, the larger the scope, the less specific are the 
results concerning educational level and program area. 

a. Please^evaluate the desirability and feasibility of conducting 

a national cost-benefit study of the following educational 
* levels: , 

Desirability Feasibility- . 

An aggregation of secondary, post- 
secondary, and adult vocational 

education programs I 

- An examination of secondary voca- 5 
ti'onal education programs only ; 

- An examination of postsecondary 

vocational education programs only 1_ — ^ 

- An examination of adult vocational 

education programs only | 1_ 

- b. For any given educational level, please evaluate the^desirabil ity 
and feasibility of conducting a national cost-benefit study which: 

Desirability Feasibility 

- Does not distinguish among program 

areas or specific programs . 

- Distinguishes among broad program 

areas only ■ \ 

- Distinguishes among specific * ^ 
programs within broad program 

areas I I 

c. For any given educational level and program area, please evaluate 
the desirability and feasibility- of ^conducting a national cost- 
benefit study which distinguishes" between the'type of, institution 
in which the training is received (e.g., community colieges, 
technical institutes, proprietary schools, on the job training, 
etc.): 

Desirability Feasibility 
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II. QUESTIONS ON MEASUREMENT ISSUES 

Numerous measurement problems will confront a study team performing a 
national cost-benefit analysis of vocational education. The following 
questions present some of the concepts that may result in measurement 
problems. 



1. One of the -first problems encountered when considering a cost-benefit 
analysis is to determine who is a vocational education student.' 
Please evaluate the desirability and feasibility of using the follow- 
ing criteria for determining a vocational education program participant 

Desirability , Feasibility 

- Enrollment in at least one vocational 
class 

- Enrollment in more than one vocational 
class „ 

- Enrollment in a fixed series of related 
vocational classes 

- Other (please specify) 



2. Once an appropriate determination has been made on what determines a 
vocational education program participant, a suitable method for 
counting thesfc students needs to be determined. Please evaluate the 
• desirability and feasibility of using the following measures of 
student participation: 

- Average Daily Attendance (ADA) 

- Average Daily Membership (ADM) 

- (ADA + ADM)/2 

- Full-time Equivalent (FTE) 

- Other (please specify) 



Desirability .Feasibility 



ERLC 



<4: 



B-14 



58 



COMMENTS 



\ 



/ 



2. 



9 

ERIC 



B-15 



53 



The costs and benefits resul ting from vocational education need to be 
compared to tho r * ,of one or more alternative activities. Those 
comparison activities* may differ by educational, level. 

a. Please evaluate the desirability and feasibility Of 'comparing 

the costs and benefits of .secondary vocational education with 

the costs and benefits of : 
_ x " Desi rabi 1 i t v Feas i bi 1 i ty 

- Attending a general education 
program 

- Attending a college preparatory 
program * 

- Not attending^ secondary school 

- A weighted average of the three 
previously mentioned activities' 

- Other (please spepify), 



■ / 

/ 

/. 

b. Please evaluate the desirability and feasibility of comparing the 
costs and benefits of. postsecondary vocational education with'the 
costs aad benefits of: 

Desirability Feasibility 

- Attending a two-year general 
curriculum college 

- Attending a four-year general 
curriculum college 

- Not attending a postsecondary f - 
. school , _ 

- A weighted average of the three 
previously mentioned activities * 

- Other (please specify) 

r 

i * 
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c. Please evaluate the desirability and feasibility of comparing 
the cost's and benefits of adult vocational education with the 
• costs and benefits of: * * 

Desi rabi 1 i ty Feasibi 1 i ty 

- Attending a two-year general curricu- 
lum college . : 

- Attending a four-year general 

curriculum college i 

- Not attending school J. :_ 

- A weighted average of the three 

previously mentioned activities ; _ 

- Other (please specify) 



The costs and benefits of vocational education accrue to various . 
individuals and groups. An essential consideration for. any cost- 
benefit calculation is to determine for which entity (lie. .an individual 
or society as a whole) costs and benefits should be evaluated in a 
national study. Please rate- the desirability and feasibility of 
evaluating, the cost and benefits accruing to the following: 

• m Desirability Feasibility 

- The vocational education enrol lee 

- Society as a whole (including the ■ 

enrollee) ^ I = 

- Society exclusive of the vocational 

enrollee - — \ — 

- Other (please specify) \ 
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A discount. rate is often utilized in cost benefit analysis to equate 
future income with present values. Please rate the desirability of 
using the following measures as a discount rate: 

Desirability 

- The prime- rate of interest * ^ . * 

- The rate of interest on gover nment 

treasury bills \ i 

- The rate of inflation N 

- Zero (discount rate^^stfould* not be 

'included in the^study)— 1 

- Other (please specify) 



The allocation of " joint costs' 1 presents a problem for cost-benefit' 
evaluators. Joint costs occur when an educational input, such as a 
teacher, piece of equipment, or school building, is used by more than 
one student group. Please rate the desirability and feasibility of 
the following treatments of joint cost: 

Desi rabi 1 i tv Fea si bi 1 i ty 

- Exclude from analysis = . 

- Evaluate the marginal cost of use : 

- Evaluate the average cost of use . 

- Other (please specify) 
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The opportunity cost 'of attending a vocatibnal, education program may 
enter* in as one of .the largest cost components in a cost-benefit 
analysis. An. opportunity cost 1s the income a student would have 
earned had he/she been working- rather than attending school. THe 
appropriate estimator of income, foregone may differ by .program level. 
Please pate the .desirability and .feasibility of the following esti- 
mators of foregone income for the secondary, postsecondary, and adult 
vocational education levels. 



a-. For secondary vocational education, the appropriate estimator of 
the. opportunity cost of attendance might be: * 

Desirability Feasibility 

- Zero, ttie individual would be 
attending school anyway < 

- The average income of individuals of 
? high school age who are not attend- 
ing school 



- A weighted average of the two 
previous measures 

- Other (please specify) 



b. For postsecondary vocational education, the appropriate estimator 
of the opportunity cost of attendance might be: 

Desi rabi 1 i ty r ~asi bi 1 i ty 

- Zero, the student would be attending 

school anyway ' 

- The avdSge earnings of individuals 
of similar characteristics who are 

not attending school , 

- A weighted average of the previous 

two estimators . : 

' - Other (please specify) 
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7. ,c. For adult vocational education, the appropriate estimator of*the . 
opportunity cost of attendance might be: * 




Zero,* the student would be attending 
school aTiyway 

- fhe. average earnings of individuals 

of similar qharacteristics who 
■/ are not attending school 

A weighted average of the two 

previous estimators 

* «t * 

-Other (please specify) 







< 








< 













8. Please. evaluate the desirability and feasibility of utilizing the, . 
-following measures of future earnings: ' 

»» * f ** * 

Desirability Feasibility , 



- Gross income (including investments) 

- Annual labor earnings 

- Individual hourly wage rates 

- Other (please specify) 
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I 

*« 

Increased earnings resulting to a vocational education graduate have 
an economic, impact greater thanHhe net. increase in the graduates 1 
earnings. This results because a large portion of the increased 
earnings will typically tie spent, increasing the income of another 
individual. Please rate the desirability and feasibility in a 
national cost-benefit study of accounting for this earnings 
multiplier effect / 

Desirability Feasibility . 



/ 



Please evaluate the desirability and feasibility of including non- 
pecuniary costs and benefits in a cost-benefit analysis, 

4 Desirability Feasibility 
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11. Please evaluate the desirability and feasibility of including measures 
of the differences in quality of vocational programs in a national 
cost-benefit analysis. 

Desirabi 1 i ty Feasibi 1 i ty 



12. A student may be enrolled in vocational education for both investment 
and consumption reasons. It is part investment because a student is 
investing in "human capital" with the anticipation of future increases 
in income. It is part consumption since a student is consuming voca- 
tional education purely for immediate personal gratification. Evaluate 
the desirability and feasibility of measuring consumption benefits of 
vocational education in a cost-benefit study. 

Desi rabi 1 i ty Feasibi 1 i ty 
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III. QUESTIONS ON DATA AVAILABILITY . 

There are several sources of data that can be used in a national cost- 
benefit study of vocational education.. The following questions consider 
-some of these alternatives. 



1. Please evaluate the desirability and feasibility of utilizing the 
following types of data in a national cost-benefit analysis ^of 
vocational education: 

_ D esirability -Feasibility 

- Existing data boses 

- Existing data bases supplemented-by 
survey data - * 

- Survey data collected exclusively for 



the cost-benefit study 



please evaluate the desirability and feasibility of using the following 
data bases in a national°cost-benefi t study: 

■ * * Not Familiar 
- " f With 
- . Desirability Feasibility Data Base 



- National Center for Educa- 
tional Statistics* (NCES) 

n Vocational Education^Data 
* "-System (VEDS) . 

- Bureau of Occupational and 
Adult Education's (BOAE) 
Statistical Reports, 1973- 
1978 

- NCES' High School and 
Beyond Longitudinal Survey 
(1980) 

- Department of Labor's 
(DOL) National LongitudjK^ 
nal Survey (1979) 

- NCES' National Longitudfi 

- Survey of the High Schooj 
Class of 1972 

- National Institute of /Edu- 
cation's (NIE) Survey/ of 
Vocational Schools in Ten 
States (1980) 
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C2. (cont'd.) 



Not Familiar 
With - ' 

Desirability Feasibility - Data Base 



*- NCES' Survey of Non- 
collegiate Postsecondary 
Students and Schools 
(1972-1980) 

- Assistant Secretary for 
Planning and Evalua- 
tion's (ASPE) Survey of 
Vocational Education 
Students,. and Teachers 
(1972) 

- Office of Civil Rights' - 
(OCR) Survey of Voca- 
tional Education Schools 
.(1979) 

- Office of Education's (OE) 
"437 Files" (Grants and 
Expenditures under State 
Administered Programs) 

- Census Bureau's Current 
Population Surveyc Supple- 
ment 

- Project Talent Data Base 

- NCES' Survey of Course 
Offerings ancf Enroll- 
ments (1973) \ 

- Survey Research^Center' s 
Youth in Transition 
Data Base (1966) 
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INSTRUCTIONS FOR DELPHI PANELISTS 

(1) Enclosed are two copies of the Delphi questionnaire. Return only one in 
the enclosed stamped and preaddressed envelope. The second is for refer- 
enece aVid to assist you in preparing your response. 

(2) Please mail your response on or before July 29 . 

(3) 'The questionnaire is divided into three sections: 

• Section I - Questions on the general design of a cost-benefit study of 
vocational education. 

• Section II - Questions concerning measurement issues and problems. 

• Section III - Miscellaneous questions. 

For all questions in Sections I and II, please rank the desirability of 

each response in order* of personal preference. Use the number 1 to 

designate the "mosfc desirable" response, the number 2 to designate the 'next 

* most desirab\le" response, etc,, until all responses are ranked . Please 

break all ties between rankings. Therefore, no two responses should be' 

assigned the same 'number. 

\ ' . 

Example : Pleas\ rank, in order of preference, the desirability of using 

the following measures to end the baseball strike: 

\ Ranking 

\ 

— Continue present negotiations between both parties 

— Send \he parties to binding arbitration - 

— Place all negotiators in a boxing ring with 

Sugar Rc*y Leonard 

Instructions for responding to questions in Section III are included 
with these questions. \ 

(4) For all questions, please, write a brief justification of your response on 
the right hand page opposite the question. This step is a critical part 
of the information collection process. 

(5) Questions in this Delphi exercise are meant to be a stimulus for thought 
on the feasibility of performing a national cost-benefit analysis of 
vocational education. You! should not feel constrained by the questions. 
In fact, you have the following options on any question: 

• Rewrite the question and answer your version if you feel . 
the original is misleading or inappropriate. 

• Suggest questions you would like to see in the next round of - 
the exercise that you feel will clarify an issue or raise a 
new alternative that the Delphi panelists should consider. 

• Write comments that relate to the question or that clarify your 
response to the question. 

The right hand page opposite each question is blank so that you can easily 
write your comments. 'All comments and justifications will be made available 
to other panelists before the Washington meeting. Anonymity will be main- 
tained in all cases. 

(6) Thank you again for your commitment to this exercise. We look forward to 
your response. If you have any questions, feel free to call Dr. Mark Shugoll 

f or Mr. Tim Helms collect at (703) 820-4350 
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QUESTIONS ON GENERAL STUDY DESIGN 

The following questions deal with general issues in the design of a 
national cost-benefit study of vocational education. 

1. A national cost-benefit study of vocational education, must be designed 
to meet the needs of its users. Please rank, in orMer of preference! 
the desirability of designing a study which would yield information 
to meet the needs of the following user groups: 

* , kc Ranking 
~ individuals , whose, needs might include determining 

\v$|iether vocational training will result in increased ' 
income* career advancement, or other benefits/ 
~ Educational institutions , whose needs mrgKl include 
increasing the efficient of vocational programs 
— Local education agencies , whose needs* might include ~~ 

securing efficient investments in vocational programs 

— State education agencies , whose n'eeds~mi"ght~include, 
determining how to distributer educational revenues * 

to maximize educational output 

•~- Federal Government , whose needs .might include 
\ allocating federal funds to the. most efficient" 

*\ alternative programs 

Other (please specify) 

\ 

\ 

I 



so 
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\Z. Please rank,inorder of preference, the desirability of each of the 
following possibilities 'in designing a national cost-benefit study 
of vocational veducation: 

Narrow the- focus of the study to a single user 
* ■ 4 and construct a compact model 
/ ~. Develop a broad and versatile model that would ' 
provide results that are meaningful to many or 
all potential users and on diverse programs' 
— , Construct several models tha;t.i.separate1y address 
the information needs of different users and 
the characteristics of different programs 
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Please rank, in order of preference, the desirability of each of the 

following considerations in designing a national cost-benefit study 

of vocational education: 

\ 4 Ranking 

— Study* design should be dictated by the current 

availability of data — : 

— Study design should be dictated by model construct 

capabil ities 

— Study design should be dictated by cost considerations . 



B-401 




COMMENTS' AND JUSTIFICATIONS 




0 



4. The scope of a national cost-benefit evaluation is of particular concern. 
The larger the scope, the more generalizable are the results. However, 
the larger the scope, the less specific are the results concerning 
educational level and program area. Please rank, in order of preference, 
the desirability of conducting a national cost-benefit study of the 
following educational levels: 



Ranking 



— An examination of secondary vocational education 
programs only 

— An examination of postsecondary vocational education 
/ programs only < 

— An examination of adult vocational educatibn programs 
only 

— An aggregated examination of secondary, postsecondary, 
and adult vocational education programs 

— An examination of secondary, postsecondary, and 
adult vocational education programs with each level 
analyzed separately 
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QUESTIONS ON MEASUREMENT ISSUES 

Numerous measurement problems will confront a study team performing a 
national cost-benefit analysis of vocational education. The following 
questions present some of the concepts that may result in measurement 
problems. 

-1. One of the first problems encountered when considenncj a cost-benefit 
9 analysis is to determine who is a vocational education^student. 
Please rank, in order of preference, the desirability 6f x using the 
following criteria for determining a vocational education^program 
participant: N *"\ J 

Ranking 

— Enrollment in at least one vocational class 

— Enrollment in more than one vocational, class 

K Enrollment, in a fixed series of related vocational 

classes ^ ~ 

— A combination of the above three measures 

Other (please specify) 



2. Once an appropriate determination has been made on what determines a 
vocational education program participant, a suitable method for 
counting these students needs to be determined. Please rank, in order 
of preference, ^the desirability of using the following measures of 
student participations 

s Ranking 

— Average Daily Attendance (ADA) 

— Average- Daily Membership (ADM) 

~ (ADA + ADM)/2 

— Full-time Equivalent (FTE) 

— Other (please specify) 



x 
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The costs and benefits resulting from vocational education need to be 
compared to those of one or more alternative activities. .Those comparison 
activities may differ by educational level. 

a. Please rank, in order of preference, the desirability of comparing 
• the costs and benefits of secondary vocational education with the 
costs and. benefits of: 

Ranking 

— Attending a general education program 

— Attending a college preparatory program 4 

— Not attending secondary school 4 . 

j A weighted average of the three previously 

mentioned activities 

— Other (please specify} 0 



Please rank, in* order of preference, the desirability of comparing 
the costs and benefits of post secondary vocational education with the 
costs and benefits of: ( ^ 5 

Ranking 

r 

— Attending a two-year general curriculum college 

— -"Attending a four-year general curriculum college 

— Not- attending a'postsecondary school 

— A weighted average of the three previously 

mentioned activities " 

— Other (please specify) 



\ 
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The costs and benefits of vocational, education accrue to various 
individuals and groups. An essential consideration for any cost- 
benefit calculation is to determine for which entity (i.e. an indi- 
vidual or society as a whole) costs and benefits should be evaluated 
in a national study. Please rank, in order of preference, the desir- 
ability of evaluating the cost and benefits accruing to the following: 

Ranking 

The vocational education enrollee 

Society as k a whole (including the enrollee) _j ^_ 

— Society exclusive of the vocational enrollee : 

— Other (please^pecify) 



The allocation of " joint costs " presents a problem for cost-benefit 
evaluators. Joint costs occur when an educational input, such as a 
teacher, piece of equipment, or school building, is used by more than 
one student group. Please rank, in order of preference, the desirability 
of the following treatments of joint cost: 

Ranking 

— Exclude from analysis , 

— Evaluate the marginal cost of use 

— Evaluate the average cost of use 

— Evaluate using game theory 

— Other (please specify) 
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III. MISCELLANEOUS QUESTIONS 

The following questions are designed to allow panelists input. in suggesting 
issues arid questions that they feel are important in designing a national 
cost-benefit study of vocational education. 

1.' Please list what you consider to,be the ^o major obstacles to performing 
a national cost-benefit analysis of vocational education: 

Obstacle I — 



Obstacle II — 



QO 
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.r, 2. Briefly. describe a strategy for overcoming, minimizing, or N dealing with J 

each obstacle listed in response to the previous question. 

Obstacle I — 




Obstacle II -- 
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Please write- two questions that/ you feel must be addressed by a research 
team in designing a national cost-benefit analysis of vocational education. 

These questions can deal with measurement problems, study methodology, 
user groups, data avail ability^ or any other issue of your choice as long 
as it has not been asked previously in this questionnaire. You do not 
have to answer the questions. 

• • . ! 

Question I — 



/ 

/ 



Questions II — 
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APPENDIX C 
RESULTS OF DELPHI ANALYSIS 
AND PANELISTS' COMMENTS 
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I. QUESTIONS ON GENERAL STUDY DESIGN 

The following questions deal with general issues in the design of a 
.„J national cost-benefit study of vocational education. 



\ 



A national cost-benefit study" of vocational education must be designed 
to meet the needs of its users- Please evaluate the. desirability and 
feasibility of designing a study which would yield information to meet 
~tfie needrofH^T^lWing user .groups: ~ 



- Individuals -, whose needs might include 
determining whether vocational /training 
will result in increased future benefits 

- Educati onal j nstltu ti ons , whose needs * 
might include making efficient investment 
decisions • i 

- Local education agencies , whose needs * 
mtjght include making program decisions 
based on local manpower needs 

- State education- agencies , whose needs 
might include determining how to 
distribute educational revenues to 
maximize educational output . 

- Federal Government , whose needs might 
include allocating scarce resources, 
among alternative programs 

- Special needs populations 
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*N = the number of panelists responding in this category 



COMMENTS ON SECTION I -- QUESTION 1 



— The proposed scale is not fine enough for my taste; so I will use + to indicate 
' . halfway between. (STUDY TEAM NOTE -- TOR EASE OF PRESENTATION TO OTHER- 
PANELISTS, A RESPONSE OF 3+ IS TABULATED AS A 3, ETC.) ... 

— Given the current political climate, it appears certain that local and state 
"agencies; will have the most critical poTroy decisions to make over the. next 
4-5 years relative to vocational educations 

— The feasibility of designing a cost-benefit sys'tem which effectively measures 
the. non-economic benefits -of vocational education remains. .doubtful irus y mind. 

— Individuals may be interested in "personal" (private") costs and benefits in 
either advancing job oportunity or in changing their career. .° 

.— Educational institutions may be interested in "program" costs and benefits in 
increasing the efficiency of vocational programs; the resulting information 

will be useful for program evaluation purposes. 

. . ' i - ■ 

— Local education agencies may be interested/ in '"'public" costs and benefits in 
.order to secure public investment in vocational programs. x 

r - Must avoid duplication. • 
■ l — Cost data at the best point ;6f usage should be enough. - " ^ 

— Considerable effort-will have to be made to identify and measure benefits 
derived from vocational education. Previous studies have been conducted 
but with little acceptance or agreement among users. 

J — Considering the economic conditions which prevail end impact upon educational 

decision makers, ^t is imperative that a national cost/benefit study address 
the n^eds of all users - from the perspective of the individual 'through the 
federal *arenai Clearly, the compelling heeds of special populations (i.e., 
handicapped, other, traditionally excluded or underepresented minorities) and. 
the efficacy of vocational education in meeting their unique needs should be 
an area of study. Current studies at the University of Illinois (Kush, 19801 
have clearly indicated the monetary/ and non-monetary benefits of vocational N 
preparation upon some of the most severely handicapped populations . . [ 
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Please evaluate the desirability and feasibility of each of the 
- - i * 

following possibilities in designing a national cost-benefit, study 

of vocational education: \ 



Narrow the focus of the study to a single 
user and construct a compact model 

Develop a broad and versatile model 
that would provide results that are 
meaningful to_many. or all potential 
users - 

Construct several models that sepa- 
rately address the information needs 
of different users 
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-Rating 
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"Please evaluate the desirability of each of the following considerations 
in designing a national cost-benefit study of vocational education:* 



Study design should be dictated by the 
current availability of data 

Study design should be dictated by model 
construct capabilities < 

Study design should be dictated by 
cost considerations 
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COMMENTS ON SECTION I ~, QUESTION 2 



- Would you focus the study efforts to "programs'," not to "users" ? 

- Would you attempt -to develop an overall framework and submodels for defining 
* .sub-components of the. general model?' 

- By breaking the study into consumer groups, tfie researchers will gain both 
depth andbreadth inrthe study; ./ 

Example - Handicapped , ■ 

Disadvantaged 
- Displaced 4tomemakers 
' - . High school dropouts 
Etc* 

- By "several models", I would "hope that you are considering. 2-4 models that 
might be focused on type of delivery system, e.g. comprehensive high, school, 
area vocational center, community college: - 

- A. broad general model can be .used as a starting point for specifications to 
J meet particular needs and interests. Moreover', construction of a narrowly 

focused model may be better achieved by specification of a general one (top 
down) by ad hoc construction (bottom up). 

May be justified only as a by-product'^of a very desirable multirmodel design 
' (Option 1.) 

tow feasibility assessment resulting from the complexity of several 
interdependent processes/procedures (i .e. / instrument development, defining 
parameters of study workscope and content, and data collection and synthesis^ 



'(Option 2). 
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COMMENTS ON SECTION I - QUESTION 3 



These are all important considerations, and they are in obvious conflict. 
Some trade-offs will be required; I regard none of them as pre-emptive . 
relative to the others. .... ... 

AIT three factors/considerations should be given equal consideration. 

, Should set the stage for, future repeats* of study, not get locked into a 
current situation 'which might produce bad or unrel-iable data. 
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The -scope of a national cost-benefit evaluation is of particular 
concern. The larger the scope, the more general izable are the 
results* However ,s the larger the scope, tha less'speci s fic are the 
results 'concerning educational level and program area. 

a* Please evaluate the desirability and* feasibility of conducting 
a national cost-benefit study of the following educational 



levels: 



An aggregation cf secondary, post- 
secondary and adult vocational 
education programs 

An examination of secondary voca- 
tional education programs only 

An examination of postsecondary 
vocational education programs only 

An examination of adult vocational 
education programs only 
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b. 



For any given educational level, please evaluate the desirability 
k a1?d feasibility of conducting a national cost-benefit study which: 

Desirability; Feasibility 



Does not distinguish among program 
areas or specific programs 

Distinguishes among broad program 
areas only 

Distinguishes among specific 
programs within broad program 
areas 



For any given educational level -and. program area, please evaluate, 
the desirability and feasibility of conducting a national cost- 
benefit study which distinguishes between the type of institution 
in which the training is received (e.g. f community colleges, 
technical institutes, proprietary schools, on the job training, 
etc . ) : 
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You may conduct a study on costs and benefits of vocational programs by: 

(1) schooi level — secondary, post-secondary, adult. 

(2) scope of area institutional basis, *local basis, state-wide and/or 

\ nationwide study. t ■ „ , 

(3) program area and/or specific program. 

l/. the programs aire* looked at separately, and then as a part of the total * 

pwgram-del4vefy-ff^eH^ 

/' ' 
The "aggregation" need not be an "integrated aggregation" although that would 

be desirable if feasible. * 

Each of the levels should be' conducted if an attempt is made. 
Should be a total - all level - but not aggregated. 

•I would encourage a cross-study analysis of relative cost-benefit measures 
across several managemenfand program content variables including promising 
strategies which lead to, effective program implementation and efficient 
distribution/consumption of resources. 
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COMMENTS ON- SECTION 'I -- QUESTION 4 b. 



— Here is another instance where trade-offs are clearly required. The more 
one asks for, the tiardei* it is to get. 

— Costs vary considerably in vocational education by specific .program area. 
' * The costs for a co-op- program are minimal when compared to a machine shop 

* program. - 

"* • .. a 

1 — The data might be easy to secu re for the entire area of vocational education 

* • but its effectiveness in the field will be. greatly aimisnea. 

— Must be defined and with some understanding among researchers and users of - 
information from study. - 
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COMMENTS ON SECTION j QUESTION 4 c. - _ 



— This is a very important component of the study. 

/ 

— .This is a must to help settle some of the arguments over whether or not 
. -secondary vocational education should, exist or not. * I 

--• Political problems — assumes same program quality arid many, items. 

' • "» • .. 
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QUESTIONS ON MEASUREMENT ISSUES ^ * 
Numerous measurement problems will confront a study 'team, performing a 
national cost-benefit analysis of ^vocational education. The following 
questions present some of the concepts that may result in measurement 
problems. ■ " ' 



,U T)neJ>£jtheL first problems encountered when eonsiderin g^Lxost^benefU — 
analysis is to determine who is a vocational education student, 
P.lease evaluate the desirability and. feasibility of us.ing the follow- 
ing criteria for determining a vocational education program, participant: 



- Enrollment in at least one vocational 
class . 

- Enrollment in more than one Vocational 

; class^ , . . 

- Enrollment in a fixed series of related" 

vocational classes 

* . - . 

- Other (please specify) 
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2. Once an appropriate determination has been made <?n what determines a. 
vocational education prolgramvparticipant* a suitable method for 
counting these students needs to be determined* Please evaluate 'the 
* desirability and; feasibility of using the following measures of 
student participation: 



Average Daily Attendance (ADA) 

Average Daily Membership (ADM) 

(ADA + ADM)/2 

Fullrtime Equivalent (FTE) 
« 

Other (please specify) : , 
(Contact Hours) 
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The feasibility varies greatly upon which statue you are talking about. 

I v.isualize a sampling procedure which can establish what V of H at least one 
fall in, each of the other .categories . Then it may be possible to use one 
detailed measurers a surrogate for all- 

Sho.yld be a vocational. program not length of time, as a -class. Various 
.occupations take different times, comparison, will not be easy, but an a 
cost study should lobk at the cost of the product produced not justNone 
segment or part of it. 

Other:' specifically designed curriculum. ^ . 
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COMMENTS ON SECTION II - QUESTION 2 



Since some funding, depends on ADA, it should be measured. Since costs relate 
to ADM, it also needs measurement and the average seems -a good statistic. 
However* FTE is perhaps- a better output measure. 

The researcher mighTwfell want:to use a. span of time as a determinant. 
If a student spends 15 Wrs -or over in- a vocational class they would be 
considered fu.ll time sttldents. 
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3. The costs and benefits resulting from vocational education need to be- 
compared to those of one or more alternative activities. Those 
comparison activities may differ by educational level. 



a. Please evaluate the desirability and feasibility of comparing 
the costs and benefits- of secondary vocational education with 
the costs and benefits of: * , 



- Attending a general education 
program 

- Attending a' college preparatory 
program 

- Not attending secondary school 

- A weighted average of the three 
previously mentioned activities 
Those special needs students attend- 
ing special education programs 

- Entering an occupation. without any 
training 

*4-. 
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b. Please evaluate the desirability and feasibility of comparing, the 
costs and benefits of postsecondary vocational education with the 
costs and benefits of: 



- Attending a two-year general 
curriculum college 

- Attending a four-year general 
curriculum college 

- Not attending a postsecondary 
school 

- A weighted average of the three 
previously mentioned activities 

- Entering an occupation without any 
training 
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COMMENTS ON" SECTION II — QUESTION 3 a, 



The largest problem with both, A and <B will be the securing of the data.*.. .. 
especially' for those youth, who are out of school. 

Major definitional problems exist with determination of general and college 
preparation curriculum students. 

Here again, the use of a weighted average may provide a good statistic, 
especially if supported with data on the three alternatives. 

I don't believe ttvere is any reason to compare with other types of ■ 
education. They all have different goals ,and expected outcomes. Maybe 
compare to training costs in other delivery systems. 

We should never make a comparison or claim of vocational education vs. 
other education without consideration of goals of individuals and all the 
variables that may enter into picture - i.e., aptitude, what if on, the 
same individual, 1 vs. 2 vs. 3, etc. 

Other: Those special needs (handicapped) students attending special 
education programs. 
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COMMENTS ON,, SECTION II - QUESTION 3 b 



For special needs populations: Potential discussion of the costs and 
benefits of integrated postsecbndary vocational education versus 
segregated rehabilitation programming* 
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c. Please evaluate the desirability and feasibility of comparing 
the costs and benefits of adult vocational education with the 
costs and benefits of: 



Attending a two-year general curricu- 
lum college 

Attending a four-year general 
curriculum college 

Not attending school 

A- weighted average of the three 
previously mentioned activities s 

Entering an occupation without any. 
training 



4 


3 


2 


1 


4 


3 


2 


1 


,-1 


1 


2 


3 


1 


2 


V 


3 


T 


1 


2 


3 




•2 


1 


3 


3 


1 


1 


2 


•2 


1 


2 


2 


1 




3 




i, 

\ 


1 


2 






1 




) 


\ 


1 







Rat i n 



i 



\ 



The costs and benefits of vocational education accrue to various 
individuals and groups. An essential consideration for any cost- 
benefit calculation is. -to, .determin e. for which ent ity (i.e. an individual 
or society as a whole) costs and benefits should be evaluated in a 
national study^. ^Please rate. the desirability and feasibility of 
evaluating the cost and benefits accruing to the following:. 



- The vocational education enrol lee 

- Society as a whole* (including the 
enrol lee) 

• Society exclusive of the vocational 
enrol lee 

- Special papulations including: ^ rural 
urban, bilingual and handicapped popu- 
lations 
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Is -this at the individual or at the societal leve"l or both? 

The data for adult, vocational students will have to be secured from othe 
sources than, those used with Secondary and Postsecondary students. This 
statement is made because the needs of adults arid their motivation for 
attending is^sp different.' 
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COMMENTS ON SECTION II — QUESTION- 4 



Perhaps it will be more broadly or alternately, defined .in the years ahead, 
but vocational- education, in some form, will continue to exist. 

Other: Potential for cost/benefit analysis for special populations including 
rural, urban, bilingual, and handicapped populations. 

I believe the benefit to society should be determined as well as the ehrollee, 
but the costs are a different question. The costs are weights against the _ 
benefits. 
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~A -discount~rate^ -U: often~uti 1 tzed~in cost benefit analysis, to. equate 
future income with present values, please rate the desirability of 
using the following measures as a discount rate: 



- The prime rate of interest 

- T^e rate of interest on government 
treasury bills 

- The rate of inflation 

- Zero (discount rates should not be 
included in the study) 



Other (please specify) 



(social dis- 
count rate) 
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The allocation of " joint costs 1 1 presents a problem for cost-benefit 
evaluators. Joint costs occur when an educational input, such as a 
teacher, piece of equipment, or school building, is used by more than 

vdhe student group. Please rate the desirability and feasibility of 

the following treatments of joint cost: 



- Exclude from analysis < 

- Evaluate the marginal cost of use 

- Evaluate the average cost of use 

- Other (please specify) (Game Theory ) 
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Feasibility 
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By social discount rate, I mean one which includes inflation and also a term 
for deferred benefits. If the analysis is done in present dollars, the 
inflation effect might be neglected. 

What if rates decrease and increase? What then? How abbut career patterns, 
change of jobs ?, 
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Allocation of joint costs is a researcha'ble issue. None of the first thre,e 
is very good. The study should generate its own model, taking into account 
some of the recent advances in Game Theory (e.g. Shapley Value, nucleolus). 

* * 

Marginal- costs would be most appropriate. if you can identify the main use 
program; Average cost would probably be easier to obtain. 



The opportunity cost of attending a vocational, education program may 
enter in as one of the largest cost components in a cost-benefit 
analysis. An opportunity cost is the income a student would have 
earned had he/she been working rather than attending school- The — 
appropriate e~sTiffi?ttJ?^ may differ by program level. 

Please rate the desirability and feasibility of the follpwing esti- 
mators of foregone income for the secondary, postsecondary, and adult 
vocational education levels, 

a. For secondary vocational education, the appropriate estimator of 
the opportunity cost of attendance, might be: 



Zero, the individual would be 
attending school anyway 



• The average income of individuals of 
high school age who are not attend- 
ing school 

- A weighted average of the two 
previous measures / 

- Other (please specify) 
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b. 



For postsecondary vocational education, the appropriate estimator 
of the opportunity cost of attendance might be: 

Desirability 



- Zero, the student would be attending 
school' anyway 

- The average earnings of individuals 
of .similar characteristics who are 
not attending school 

- A weighted average of the previous 
two estimators 

- Other (please specify)^(W?ighted 
average and comparison with costs 
of other education programs) 
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COMMENTS ON SECTION IT QUESTION 7 b. 
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7. c. For adult vocational education, the appropriate estimator of the 
opportunity cost of attendance might be: 



- Zero, the- student would be attending 
school anyway 

- The average earnings of individuals 
of similar characteristics who 

are not attending, school 

- A weighted average of the two 
previous estimators 

- Other (please specify) : 
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8* Pjease evaluate the, desirability and feasibility of utilizing the 
following measures of future earnings: - & 



- Gross income (including investments), 

- Annual Tabor earnings 
-Individual hourly wage rates 

- Other/(please specify) 
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COMMENTS ON SECTION II - QUESTION 7 c. 

■ ' ~ . * 



Characteristics of students will vary so much, data;* will be meaningless-. 





COMMENTS ON SECTION II ~ QUESTION 8 



I assume that all of these will be discounted to present" values and will 
take account of ages of earners* 

Will be difficult to obtain ♦ 

Hourly income could be used with the assumption that an individual is employed 
full time and can work full time* 
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9. Increased earnings resulting to a vocational education graduate have 
an economic impact greater ^than the net increase in the graduates' 
earnings. This results because a large portion of the increased 
earnings will typically be spent, increasing the income of" another 
individual. Please rate the desirability and feasibility in a 
national cost-benefit study of accounting for this earnings 
multiplier effect . 

Desirability Feasibility 
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10. Please evaluate the desirability and feasibility of including non- 
pe cuniary costs and benefits in a cost-benefit analysis. 
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COMMENTS ON SECTION II - QUESTION 9 



This is not a transparent benefit and to use this concept will require con 
siderable study . The basic economics of exchange iroply that the worker 
values his wage received^more than his time spent and that the employer 
values the. work done more than the wage paid. 
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COMMENTS ON SECTION II -- QUESTION 10 



This is very important and involves evaluating trade-offs , between dollars 
a'nd quality of life. Multicriterion benefit-cost models are beginning to 
emerge and should be looked into. 

This is perhaps the most difficult aspect of "this study to deal with. 



C-32 



128 



^9 



11. Please evaluate the desirability and feasibility of including measures 
of the differences in quality of vocational programs in- a national 
cost-benefit analysis. 



Desirability Feasibility 
4 3 211 48312-1 T 
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12. A student may be enrolled in vocational education for both investment 
\ and consumption reasons. It is part investment because a student is • 
investing in "human~capital" with the anticipation, of future increases 
in income. . It is part. consumption since a student is consuming voca- 
tional education purely for immediate personal gratification. Evaluate 
the desirability -and feasibility of measuring consumption benefits of 
vocational education in a cost-benefit study. 
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COMMENTS ON SECTION II — QUESTION 11 



~ The potential political reactions to such measures make this a delicate 
4 matter - It is worth consideration but including such measures may lower 
* acceptability of the whole effect. 

— Careful consideration of standard .criteria for quality or effectiveness _ 
must be established. 
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This -area is very important, though probably very hard to collect data on. 
Many detractors of vocational education cite personal consumption as a waste, 
yet 'it is almost impossible to factor it out. The detractors feel the programs 
are^'too expensive to run so someone can learn how to fix their car or some 
other personal skill. These data could help to refute this. 



- This is closely related to 10. 
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QUESTIONS ON; DATA AVAILABILITY- 

There are several sources of data that can be used in. a national cost- 
- benefit study of vocational education. The following questions consider 
some of these alternatives. * . 

1.. Please evaluate the* desirability and feasibility of utilizing the 
following types of data in a national cost-benefit analysis of 
vocational education: 



ERJC 



Desirability Teasibili 



Exi sting data bases " 

Existing data bases supplemented by 
survey data * 

Survey data collected exclusively for 
the cost-benefit study 
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2. Please evaluate the desirability and feasibility of using the following 
data bases in a national cost-benefit .study: 



Not Familiar 
With 



- National Center for Educa- 
tional Statistics' (NCES) 
Vocational Education Data 
System (VEDS) 

- Bureau of Occupational and 
Adult Education' sjBOAE) 
Statistical - Reports, 1973- 
1978 

■ - NCES' High School and 

Beyond Longitudinal Survey 
(1980) 

- Department of Labor's 
(DOL) National Longitudi- 
nal Survey (1979) . 

- NCES' 'National longitudinal 
Survey of- the High S chool 

cra•s^r^Dff^l972 , — 7— 

- '- National Institute of Edu- 
cation's (NIE) Survey of 
Vocational Schools in Ten 
States (1980) 
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COMMENTS ON SECTION III --• QUESTION 1 



The major .reliance should be on existing data bases but there may be gaps 
which require- survey data. • — 
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\ .... 
COMMENTS ON SECTION III— QUESTION 2 ,'" [ . .. • 

— You may need a cross-checking of these data bases for verifying data needed- 
/for a cost-benefit study. : — : 

— Recommend that you conduct your own sample - could use existing data bases - 
to draw sample - under these cirucumstahces a higher rating could be made on 
some of the data bases listed. The VED's System would probably be best for this 
use on vocational students. You would need another base for more vocational 
students from one of those listed of which I am not knowledgeable. 




-2v(cbht'di) 



Not Familiar 
Hith 



NCES' Survey of Non- 
collegiate Postsecondary 
Students arid Schools 
(1972-1980). 

Assistant Secretary for 
Planni rig and 'Eva 1 ua- 
tion/s(ASPE) Survey of 
Vocational Education 
Students and Teachers"" 
(1972) 

Office of Civil Rights' 
(OCR) Survey of V c oca T 
tional Education Schools 
'(1979) 

Office of Education's (0E) 
"437 Files 11 (Grants and 
Expenditures under State" 
Administered Programs) - 

Census Bureau's Current 
Population Survey Supple- 
ment 

Project^Talent Data Base N 

NGEST! Survey of Course 
Offerings and Enroll- 
ments (1973) 

.Survey Research Center's" 
Youth in Transition 
Data Base (1966) 
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ROUND TWO . - • - - 

I. QUESTIONS ON GENERAL STUDY DESIGN . * ' 

The following questions deal with general issues, in the. design of a 
national cost-benefit study of vocational education; 

» 

1. A national, cost-benefit study of vocational education must be designed 
to meet the needs of its users. Please rank, in order of preference, 
<the desirability of designing a study which would yield information 
to meet the needs ot the "following user groups: 

Mean * 



*Mean indicates the mean value assigned to this option by panelists, 



Individuals > whose needs 'might include, determining 

whether vocational .training will result in increased 

income, career advancement, or other, benefits 3.8 

Educational institutions / whose needs might include 

increasing the ' efficiency of vocational/ programs 3.7 

Local, education agencies , whose needs might include , 

securing efficient investments. in vocational programs 3.2 

Suate education agencies , whose needs might include 

determining ihow to distribute educational revenues j 

to maximize educational- output 2 -P 

Federal Government , whose needs might include 

allocating/federal funds to the most efficient 

alternative programs 2.3 

Other (please specify) * - 



COMMENTS ON SECTION I - QUESTION 1 



Dissemination of relevant information to individuals is important, but 
I feel that for them there are better approaches than benefit-cost models, 
except perhaps at the nominal level. The other four groups have comparable 
needs and the shadings between the ratings 1-4 are small compared with the 
differences w"th -individual- needs (5). i.e. 

■ i » 

high \ [U - ; 1 ^t: 

The federal government's needs would entail the use of cost-benefit 
data* in judging;the return on .Investment. It may be necessary or desirable 
to, fund programs which have both high costs and high benefits. Funding > 
decisions might not always favor the most efficient alternative programs.. 

Focusing upon meeting, needs of local- education agencies, a wst-benefit 
study shall be made, then' the study can be expanded toward meeting other 
.agencies' 1 needs. / v ' 

I would target on my first (state education agencies) and second (Federal 
Government) rankings. - t 

The 'RFP should have .-made clear that the main 'purpose of any resulting 
national cost/benefit study' was to serve federal policy determination 
heeds. 

The usefulness of any program ratios at less than national policy levels 
is debatable, given the likelihood that these Vatios will vary considerably 
across states, communities, and institutions. 

-the effectiveness and ultimate impact of study measures/findings will be 
determined according to its utility for those most directly involved in 
the provision and consumption of vocational services. Thus educational 
institutions, individuals and LGA's are the user groups of primary import 
in the development of any national cost-benefit study. 

Given the- recent White House economic victory in Congress, an emphasis on 
the needs of state education agencies would appear to be the most appropriate 

The ranking is based upon (1) the individuals needs being met. The 
remaining ranking is based upon the funding sources and their understanding 
• of true need 
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Please rank, in order of preference, the desirability of each of the 
following possibilities in designing a national cost-benefit study 
of vocational education: 

"Mean 

— Narrow the focus of the study to a single user 

and construct a compact model - 2.8 

— Develop a. broad and versatile model that would 
. provide results that are meaningful to many or 

all potential users and on diverse programs 1 «8 

Construct several models that separately address 
the information needs of different users v and 

the characteristics of different programs 1.4 



Please rank,/in order of preference, the desirability of each of the 
following considerations in designing a national cost-benefit study 
of vocational education: 

Mean 

-- Study design should be dictated by the current . 

? l 

availability of data zl 

— Study design should be dictated by model construct, 

ca pabil \t i es * Li 

— Study design should be dictated by cost considerations - 2.4 
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COMMENTS ON SECTION I - QUESTION 2 



A good broad model can be specialized to achieve the goals 'of the other 
two statements. However, if a single user is paying for the model, he 
has a right to expect it to be focused on his needs. 

The most useful and productive approach may be option 3 with some degree 
of data compatibility to assure that outputs can be aggretated across 
models for different users, e.g. community colleges, AVC's, etc. 

I do not think it is possible to design a single model which will serve 
all potential users within realistic cost constraints. 

While the basic procedures for determining cost and benefit might ,be 
the same (or similar) regardless of level of aggregation (national, 
state, or local), the procedures for program selection (universe or 
sample) and the approach to data acquisition would vary considerably. 

The model should be practical rather than theoretical. An elegant model 
which cannot be applied is of no use, in my opinion. 

The development of several discrete models will provide the necessary 
b>«adth and depth in addressing the critical areas of assessment - i.e. 
the dimensions related to multiple 'educational levels and varied user 
groups. Strategies for the analysis of common program elements (which would 
be available through the broad and versatile model design) should be 
incorporated into the multiple model study design. 

Only by developing a number of models is it possible to take into con-^ 
sideration the myriad of diverse needs of the many potential users. "A 
broad and versatile model" would provide severely limited information. 

A broad model "would allow many individuals and agencies to use the 
information. 
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- If the design requires expenditures beyond the project budget, it has no 
chance to have a good outcome; , However, once budget feasibility is establi- 
shed, the model should not be strictly limited by currently available data 
-since one major benefit^of a b-c model is identification of data needs. 

-f Since ajiational cost-benefit study has yet to be developed for/by 
vocational education, it is not likely that the currently available data 
will be adequate or appropriate. 

- Given that 1) current data availability and potential resources for the 
study pose severe programmatic constraints, and 2) the quality of study 
activities and findings are dependent upon a solid, comprehensive model 
design, the consideration of model construct capabilities are paramount. 
Of course the delimiting factors cited in point 1 (above) will necessitate 
flexibility in the development of the model. 
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The scope of a national cost-benefit evaluation is of particular concern. 
The larger the scope, the more general izable are the results. However, 
the larger the scope, .the less specific are the results concerning 
educational level and program area.; Please rank, in order of preference, 
the desirability of conducting a national cost-benefit study of the 
following educational levels: , 

Mean 

i 

An examination of secondary vocational education 
programs only 

— An examination of postsecondary vocational education 

programs only ■ 3 '1 

~ An examination of adult -vocational education programs 

only , * . 

— An Aggregated examination of secondary, postsecondary, 

and adult vocational education programs 3.9 

— An examination of -secondary, postsecondary, and 
adult vocational -education programs with each level 

analyzed separately — 
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COMMENTS ON SECTION I — QUESTION 4 



Since vocational education is a relatively large and diverse national 
enterprise, it would be most helpful to examine all levels independently. 
" If one had to choosebetween the three levels, however, the preference , 
should, go to the secondary level because it. represents the largest enroll- 
ment and resource consumption. * x 

— The types of benefits differ considerably by institutional level. For 
example, while job placement rates and earning levels might be the most 
appropriate benefit measures for postsecondary and adult programs, the 
-benefit of secondary programs might be most appropriately judged by levels 
of skill proficiency or attitudinal changes. Consequently, I do hot see 
how an aggregate benefit assessment across institutional levels could be 
fairly constructed. 

The postsecondary and adult option (combined) was selected second because 
I feel- traditional cost/benefit analyses are most easily applied at these 
levels than at the secondary level. ^ 

Separate analysis of vocational edyeation programs will allow for the 
specificity required for a valuable, cost-benefit study. However, a cross 
study analysis of several relative cost-benefit measures across several 
management and program content variables is encouraged. 

— A national cost-benefit evaluation restricted to a particular program level 
or aggregated aver all levels would be of little use. 

— The #1 ranking would allow, for the. generation of both general and specific 
data. 
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QUESTIONS ON MEASUREMENT ISSUES 

Numerous measurement problems will confront a study team performing a 
national cost-benefit analysis of vocational education. The following 
questions present some of the concepts that may result in measurement 
probl ems . 

1. One of the first problems encountered when considering a cost-benefit 
analysis 1s to determine who is a vocational education student. 
Please rank»in order of preference, the desirability of using the 
following criteria for determining a vocational' education program 
participant: 

Mean 



Enrollment 1n at least one vocational class 2.9 
Enrollment 1h more ';han one vocational class 2A 
Enrollment 1n a fixed, series of related vocational 



classes 



1.8 



A combination of the above three measures- ?-.Q 
Other (please specify) . * 



Once an appropriate determination has been made on' what determines a 
vocational education program participant,- a suitable method for 
counting these students needs to.be determined. Please rank, in order 
of preference, the desirability of using the following measures, of 
student participation: 

Mean 



Average Daily Attendance (ADA) _2JL 

AverageDaily Membership (ADM) . 3 ,- 3 
(ADA + ADM)/2 

Full-time Equivalent (FTE) „ -Li- 
Other (please specify) . 
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COMMENTS ON SECTION II - QUESTION T 



At the post-secondary and adult levels .clearly the interest lies with 
students training for a specific occupation. However, at the secondary 
_level where the purposes and benefits of vocational education partici- 
pation can be more broadly construed, attention should also be paid 
to students in exploratory vocational education programs; That is t 
to say care must be taken at the secondary level in distinguishing 
between those students in occupational specific programs and those stu- 
dents in exploratory or prevocational programs. Different benefit measures 
should be applied. In any case, the benefits should not be averaged. 

if one distinguishes grades of vocational education students as provided 
in II-2 below, then I would change the ratings'to 1, 3, 4, 2, with ■ 



Nearly all states have established vocational program course sequences 
through which students >accumul ate skills over a period of 1-2 years. 



You have to define the level of programs - class level, course level, 
or program level. Personally, I would like to suggest the level of 
program for a national study. 

I would very. much have liked to have seen a. question related to a program 
rather than a class. If this was available under other , I would .have 
rated i t #1 . 

FTE provides for the most accurate measurement of student participation 
time in n a vocational "program. 
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— Clearly student contact hours in the program is the preferable approach to 
measuring -program participatipn. However, an ADA or ADM count combined with 
a sorted enrollment count (II above) would be satisfactory. . . 

— FTE is an excellent measure of load on the system" (I assume this is based 
on some equivalence with credit hours- of enrollments.) • However, serious- 
ness of paetfci pants is measured by average daily attendance... I suggest 
for a class^ie measure:- * ■* 

(No. of hours per week) X (No. of enroll ees) X R 

where R is a reduction. factor to account for absentees. R should probably 
not be linear. 

— " Use one criteria to be universally adaptable! . 

-~ Other Some other indicator measuring participation except attendance~or 
membership.. ADA and ADM are completely unsatisfactory to me as an indicator 
of a participant. 

— Since instructional service outputs are for students', the more appropriate 
unit for study would seem to be a measure of student service unit. The 

' more precise and widely applicable cost unit appears to be the concept of 
the FTE student, based- on ia standard number of student contact hours. 
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3. The costs and benefits resulting from vocational education need to be 

compared, to those of one or more alternative activities. Those comparteoh- 
" activities may differ by-educational level. 

a. Please rank, in order of preference, the desirability of comparing., 
the costs and benefits of secondary vocational education with the 
costs and benefits of: 

,\ Mean 



Attending a general education program __L§_ 

Attending a collide preparatory program 2 » 3 

Not attending secondary school 2 » 8 
A weighted average of the three -previously 

mentioned activities 2 » 7 

Other (please specify) 



b. Please rank, in order of preference, the desirability of comparing 
the costs,. and benefits of postsecondary vocational education with the 
costs and benefits of: 



Mean 



Attending a two-year general curriculum college 2 '° 

Attending a four-year general curriculum college ■ 3.1 

Not attending a postsecondary school 2J 
A weighted average of the three previously 

mentioned activities 2>7 

Other (please specify) 
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COMMENTS ON SECTION II — QUESTION 3 a. 




It is not clear to me -why a comparison- of the vocational educational 
cost/benefit ratios to those 'of the ratios discovered for other, 
programs is necessary or particularly valid-particularly in terms of the 
college preparatory program. I suppose if one "discovered that-general 
education^ students (with careful, matching of ability and SES ..background) 
did about as well as Vo Ed" students" in terms of employment and wages at 
substantially lower programs costs one might use the information for ^lo- 
cation decisions. However.^I'm not at all comfortable with where such 
ratios comparisons might lead policy makers. Present cost/benefit ratio 
studies assign much higher values to elementary and junior high 
education than to the higher grades. Similarly lower education scores 
-higher than higher education. . What are we to make of this in terms of 
public policy? : • 



-rents and students alike will find information on the costs and benefits 
orvvocational education most helpful when trying to judge its valud relative 
to\olle'ge prep programs. 

Hew .would you give a different weight to three areas? 
Other -rttnV non-public -voc. programs, CETA programs. I do not believe 



the stuoy\shoYld compare academic and voc. ed. 



V 
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COMMENTS ON SECTION H ^ QUESTION 3"b. 



/ / • 
One might want to compare the relative effectiveness and efficiency of 
vocational eguca^ion training at the post-secondary and secondary levels. 
Although thjs v^ould be a dif^ult task to do fairly. 

Thfi most appropriate comparison would be between two populations with 
similar occupational goals - one of which participated in post-secondary 
Vo Ed arid the other which did not. * • 



~ Here again, such information will be extremely useful in career planning 
for potential post-secondary students. 
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The costs and'benefits of vocational education accrue. to various 
individuals and groups. An.esseritial consideration for any cost- 
benefit calculation is to determine for which entity (i.e.- an indi- 
vidual or society as a .whole) costs and benefits should-be*evaluated 
in a national study. Please rank, in order of preference, the desi»> 
ability of evaluating the cost and benefits accruing to the "following: 

* * ' C ,Meah 

— : The vocational education enrollee - ■■ liS 

Society as a whole (including the enrollee) LL. 

— Society exclusive of the vocational enrollee 3 «0' 
Other -(please specify) 



The allocation of " .joint costs " presents a problem for cost-benefit 
evaluators.' Joint costs occur when an educational input,,, such as a 
teacher, piece of equipment, or school building, is used by more than 
one student group. Please rank, in order of preference, the desirability 
of the following treatments of joint cost: 

Mean 

— Exclude from analysis — ~u- — Li 

— Evaluate the marginal cost of use — <Lil 
-- Evaluate the average cost of use - — hi 

— Evaluate using game theory - — hi 

— Other (please specify) 

i 
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COMMENTS ON SECTION II - 



- QUESTION 4 



The primary concern of the study ^should be with individual benefits by 
program. j! 

i ...•*•" \ 
This rates the "who benefits" "who pays" question 



V 



Both society as a whole and the individual enroliee must have net positive 
gain (benefit-cost) jfor the program jto be workable. 

Difficulties exist w<ith measurement of j the non-economic costs and benefits, 
associated with vocational education;, e.g. increased levels of employee/ 
worker satisfaction! < . ! " . \ \ 

■ • . ■ ■ i 

Other : Special populations, including rural, urban, bilingual, and handi-? 
capped populations. j \ 
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COMMENTS ON SECTION II - QUESTION 5 



— In measuring costs, the model should attempt to strive for reasonable . 
'• precision where such precision is likely to make a, significant difference 
• _ in cost calculations. For instance a. classroom in which distributive 
-- education is taught will vary little in construction costs from a regular 
' classroom (averages would be appropriate). On the other hand the costs 

of constructing a heavy machine shop should not be averaged with total 

building costs. . 

I don't understand the- f point of the question when it speaks to teachers. 
I see no problem with using average teacher salaries given a single 
sa.lary schedule (if that is what is meant). If a teacher splits his time 
between Vo ? Ed and the general curriculum,, then his salary should be 
prorated according to program assignment. 



! 



-- Other -: judicious use' of all. three methods. For starting a new added 
program, marginal costs may be the best; for evaluating a whole system, 
' average cost is attractive; game theory methods are relevant when con- 
sidering several- different added programs or combinations thereof. 
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(The following questions are designed to al-low panelists input in suggesting 
issues and^ questions that they feel are important in designing a national 
cost-benefit study of vocational education.) 
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Please list what ypu consider to be the two major obstacle's to performing 
.a national cost-benefit analysis of vocational education: 

Obstacle I The most important obstacle is defining and identifying the 
control groups from which the marginal benefits of vocational education 
•training can be (jerived. ^ 

Obstacle II Constructing operational measures of the benefits of the 
non-occupational specific secondary, vocational education programs and 
translating these into monetary values* ^ 

Obstacle III Obtaining valid employment and wage histories of students; 
Comments : 



2. Briefly describe a strategy foAovercoming, minimizing, or dealing with 
each obstacle listed in response to the previous question. 

Obstacle I 



Obstacle II 



Comments : 
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Please list what you consider to be the two major obstacles to performing 
a national cost-benefit analysis of vocational education: 

Obstacle I Lack of good measures for non-monetary costs and benefits. 



Obstacle II Critical gaps in the data base. 



Comments : Observations of the national, state, and other decision-making 
leads to the conclusion that political and quality of life factors play a 
crucial roles. Hence, models which neglect these or dismiss them with" the 
disclaimer that "since they con't be measured, we will omit them from our 
model" are seriously deficient. 

One feature of model building is that a good 'theoretical model helps pin- 
point what the crucial data needs are. It usually is the ease that some 
of the needed data has never been collected, tabulated or stored (in acces- 
sible formH 

Briefly describe a strategy for overcoming, minimizing, or dealing with 
each obstacle listed in response to the previous question. 

Obstacle I The Rice-TIRR group has focused on handling non-monetary 

costs and benefits in rehabilitation. Some of their results seem applicable 

here. (Most of their reports are in the REHAB files.) 



Obstacle II Several approaches are: a-rset up an MIS (Management Infor- 
mation System) to process and'handle such data as are available; b) encourage 
adding important items to statutory reporting systems; c) find acceptable 
surrogates which are available. 




Comments: 
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Please list what you consider to be the two major obstacles to performing 
a national cost-benefit analysis of vocational education: 

< 

Obstacle I Developing a model or series of models which will meet the 
Expectations and needs of the diverse and numerous user groups. 



Obstacle II Suggesting ways to measure the non-economic costs and benefits 
or vocational education. 



Comments : 



Briefly describe a strategy for overcoming, minimizing, or dealing with 
each obstacle listed in response to the previous question. 

Obstacle I Once a model or series of models is developed, a series of 
rigorous field tests with each of the major user groups from a representa- 
tive sample of states should be conducted. The further development and 
evaluation of the models involve a sizable group of vocational education 
leaders over an extended period" of time (2-3 years) to assure. 

Obstacle II Continue to review the literature and discuss this issue 
with knowledgeable individuals. 



Comments : 
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Please' list what you consider. to be the two major obstacles to performing 
a national cost-benefit analysis .of vocational education: 

Obstacle I The quality of data need a unified data system including 
common definitions of data'items and systematic data collection. 

Obstacle II Finding group comparable with others. 

f 

O bstacle III Sensitivity of information opening to the "public". 

Obstacle IV Need a cooperative participation of selected agencies and 
• individuals. * * . 

Comments: 



Briefly* describe a strategy for overcoming, minimizing, or dealing with 
each obstacle listed in response to the previous question. 

Obstacle I M£y need a longitudinal approach to data definition and 
collection. „ 

Obstacle II A unified data system 

Obstacle III and IV A cooperative participation of selected agencies. 



Comments: 
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Please list what you. consider to be fhe two major obstacles' to performing 
a national cost-benefit "analysis of vocational education: 

Obstacle I Determining the specific cost per program and, then aggregating 
to represent a, cost of Vocational Education. Both Direct and Indirect costs. 



Obstacle II Getting^ an agreement on what benefit to measure and then 
measuring the benefit or benefits . t 



Comments : There exists little information today in regard to cost per program. • 
One can get from existing reports expenditures from local, state, and federal 
levels, but this is probably not a good indication of actual cost because ,so 
many variables are related that may not'be directly related to, training. A few 
years ago we developed standardized cost per program and I can tell you it is 
a lengthy exercise;' When you start prorating over secondary, postsecondary, and" 
adult it is even' more complicated. 

I-assure you that an agreement cannot be reached .on benefits. In-'the study that 
was conducted here in Oklahoma we attempted this. We ended up with six objective 
functions: (continued on next page) , 
Briefly describe a strategy for overcoming, minimizing, or dealing with 

each obstacle listed in response to the- previous question. 

Obstacle I Narrow down exactly what kind of cost you are going to use and 
represent your study as a study that has been conducted under this specific 
set of assumptions. 



Obstacle II The same as the above comment. Acceptance of benefits are 
extremely difficult to get. You may want to measure benefits under different 
alternatives. 



Comments : A constraint that you need to be aware of is the fact that you 
do not have an unlimited supply of individuals that can enter any level of 
employment that they choose. You really have a supply of persons that have 
varying attitudes- and abilities and the benefits derived from vocational 
training may be -a great return to cost if this was considered. 

Based on previous research done in this unit we had access to 40,000 sophomores, 
10,000 seniors, and 10,000 adults' GATB scores to use as an indicator with 
supply. When trying to fill jobs we found that certain occupations competed 
for the same GATB scores and that our supply of individuals were not available 
to satisfy all the job vacancies. 
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' 1, Comments (continued): 

- 1. maximize entry level wages; 

2. maximize- supply; * . 

3, « maximize returns to taxes; . ; - : . 
4* maximize to fill demand for trained workers; 

'5. maximize number of* students served; 

6. minimize costs* , 

If you cannot get agreement on benefits to- be measured then no one will 
...accept your study.' .Therefore-, 1 assure you that you have v a challenge. 



\ 



7 i5s 

ERIC C-63 



.Please list what you consider to be the two -major obstacles to performing 
a national' cost-benefit analysis of vocational education: 

Obstacle I Lack of common definitions of what a Voc. Ed. program is- in 
the various states. 



Obstacle II Lack of complete' fiscal information at' many levels. 



Comments: 



Briefly describe a strategy for overcoming, minimizing, or dealing with 
each obstacle listed in response to the previous question. 

Obstacle I Set a standard for the study only. • - 



Obstacle II Collect the data. 



Comments: 
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1. Please 11st what you consider to be the two major obstacles^ performing 
a national cost-benefit analysis of vocational education: 

Obstacle I The inclusion of non-pecuniary costs and benefits in the study . 



Obstacle II The consideration of the impact of vocational programming upon 
special populations (i.e. rural, urban, bilingual, and- handicapped) . 



Comments : This is probably ';he most difficult measurement area , upon which 
to gain consensus. However, it is an area in which vocational education 
stands to promote its most compelling justification for existence - social 
(as well as economic) benefits. - 

This consideration is particularly critical to-federal and state adminis- 
trations, as a result of set aside requirements for special populations. 



,2. Briefly describe a strategy for overcoming,: minimizing, or dealing with 
each obstacle Listed 1n response to the previous question. 

Obstacle I If the study team selects the method of measuring non-pecuniary 
benefits according the monetary values, an additional Delphi procedure will 
be required to determine the variables to be selected and the weighting of 
such variables. 

Obstacle II The study team may wish to devise strategies for the collection 
of special population data from 1) segregated vocational programs, and 
2) integrated, regular vocational programs which include representatives 
from identified special populations. Data Collected for these populations 
should be equivalent to the data collected for the general study populations 

Comments: 
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.Please list what you consider to be the two major obstacles to' performing 
a national cost-benefit analysis of vocational education: 

Obstacle I Securing clear accurate data that is transferable through- 
out the U.S. 



Obstacle- II Money and time. A study of this nature should have the 
time and resources to develop a research model or models that will be 
researchable and yield the data needed to be comprehensive cost- 
benefit study. 

Comments: 



Briefly describe a strategy for overcoming, minimizing, or dealing with 
each obstacle listed in response to the previous question. 

Obstacle I Develop several methods of gathering data based upon the 
uniqueness of states. 

Obstacle II With the current mood in Washington, about the only hope 
is time to do the job. Money will not be forthcoming., > 

Comments: 
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Please wri€e two questions that you feel must be addressed by a research 
team in designing a national cost-benefit analysis of vocational education. 

These questions can deal with measurement problems, study methodology, user 
groups, data availability, or any .other issue of your choice as long as 
it has not been asked previously in this questionnaire. You do not have 
to answer the questions. 

Question I How. can appropriate managers be encouraged to use benefit- 
cost methodology and results? ■ - 



Question II ~ Whe v re does one draw the line between inputs, by the model 
builder and inputs from the responsible manager? 

My thesis is\ that value judgements should be provided by the manager at 
as near as possible to the time he needs to make a decision. 



Comments : One clue to this question is documentation. I mention two 
important 'levels: 

A. Technical documentation 
' B. User documentation 

A criterion for satisfactory technical documentation is transferability , 
i.e. a knowledgeable user should be a61e to understand (a) the definitions 
of all of the variables, (b) the data sources needed, (c) the underlying 
assumptions and.(d) the logic supporting -the equations well enough to apply 
the model without recourse to the model builder. 

Relatively few models paid for fay the U.S. Government meet this criterion. 
However, the Dept. of Energy now requires archival storage (at its Argonne 
Laboratories) in transferable form. 

User documentation, is even rarer than good technical documentation. What 
I have in mind here includes verbal formulation of all equations, assumption 
etc. so that a manager can understand the thrust of the model even though 
he is not a specialist (i.e. knowledgeable) in model building. 

(If this question is followed up, I can^rovide a number of references.) 
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3. Please write two questions that you feel must.be addressed. by a research- 
team in designing a national cost-benefit analysis of vocational education. 

These questions can deal with measurement problems, study methodology, user 
groups, data availability, or any other issue of your choice as long as 
it has not been asked previously in this questionnaire. You do not have 
to answer the questions. 

Question I -r What data are currently collected by states or are available 
from national sources (e.g. NCES) that could be analyzed-and used as gross 
indicators of the costs and benefits of vocational education for various 
special needs populations? > _ 



Question II — Why should vocational educators at all levels be concerned 
with cost-benefit analysis? 



Comments: 
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Please write two questions that you feel must be addressed by a research, 
team in designing a national cost-benefit analysis of .vocational education. 

r t m 

I * 

These, questions can deal with measurement problems, study methodolpgy^user 
groups, data availability, or any other issue of your choice as long as 
it has not been asked previously-in this questionnaire. You do not have 
to answer the questions. 

Question I — Att^ t to analyze preliminary data' available to each of the 
selected local agencies/ 



Question II 



Comments : 
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Please write two questions that you feel must be addressed by a research 
team in designing actional cost-benefit analysis of vocational education. 



These questions can deal with measurement problems, study methodology', user- 
groups, data availability, or any other issue of your choice as long as 
it has not been asked previously in this questionnaire. You do not have 
to .answer the questions. 

Question I — Are you planning to use aggregate data or individual pro- 
gram data? 



Question II — How are you planning to get information to study cost-benefit 
-if aggregate data are not used? 



Comments: 



1 /?tr 
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3, Please write two questions that you feel must be addressed by a research 
team in designing a national cost-benefit analysis of vocational education. 



These questions can deal with measurement problems, study methodology, user 
groups, data availability, or any other issue of your choice as long as 
it has not been .asked previously in this questionnaire. You do not have 



Question I .-- What are the non-pecuniary benefits derived from vocational 
education (i.e. secondary, post-secondary, and adult programming)? 



Question ,11 — What have been the economic and non-pecuniary benefits of 
vocational programming (i.e. secondary, post-secondary, and adult) on 
^special populations (i.e. rural, urban, bilingual , and handicapped)?, 



to answer the questions. 



/ 




Comments : 



/ 
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Please writeNtwo questions that you feel must be' addressed by a research 
team in design^ a national cost-benefit analysis of vocational education. 

' \ 

These questions can deal with measurement problems, study methodology, 
user groups, data availability, or any other issue of your choice as long 
as it has not been asked previously in this questionnaire. You do not 
have to answer the question. 

•Question I How are-special needs students being accomodated in vocational 
education and how does the cost of their involvement- affect the program 
within which they are enrolled? 

\ 



Question II What is the payback period (taxes and oon receipt of welfare) 
for graduates of vocational programs as opposed to non-vocational graduates? 



Comments: 



4 
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APPENDIX D 
LIST OF PARTICIPANTS AT 
THIRD ROUND DELPHI CONFERENCE 
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CONFERENCE ON THE FEASIBILITY OF CONDUCTING A 
NATIONAL COST-BENEFIT ANALYSIS OF VOCATIONAL EDUCATION 



August 10, 1981 



9 A.M. - 5 P.M 



LIST OF PARTICIPANTS 



Dr. Ralph Bregman ; 
National Advisory Council for Vocational Education 

Ms. Barbara Dunn 
Youthwork, Inc. 



Dr. George Hagerty 

U.S. Department of Education 

Division of Personnel Preparation 

Dr. Paul Hippolitus 

President's Commission on Employment of the Handicapped 

Dr. Krishan Paul 

American Vocatfonal Association 



Dr. L: Allen Phelps 

Department of Vocational and Technical Education 
. University of Illinois 



REHAB GROUP, INC. STUDY TEAM 

Dr. Diane Simison - Project Director 

Dr. Mark Shugoll - Principal Investigator 

Mr. Tim Helms 

Ms. Dorine Seidman 

Dr. David Rodney 
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CONFERENCE ON THE FEASIBILITY OF CONDUCTING A 
NATIONAL COST-BENEFIT ANALYSIS OF VOCATIONAL EDUCATION 



August 10, 1981 9 A.M. - 5 P.M. 

AGENDA 

9:00 Continental Breakfast 

9:20— Introductions - - 

9:30— Overview of the Rehab Group, Inc. study effort 

9:45 - 10:45 Discussion: Evaluating the merits and parameters of a 

national study 

10:45 - 11:00 Break 

11:00 - 12:00. Discussion: Identification and measurement of vocational 

benefits 

12:00 - 1:30 Lunch 

1:30 - 2:30 Discussion: The availability and quality of data on 

vocational programs and vocational students 

2:30 - 3:30 Discussion: Issues identified by conference participants 

3:30 - 3:45 Break © 

3:45 - 4:45 Discussion: Conclusions on the overall feasibility and 

utility of a national cost-benefit study 

4:45 - 5:00 —-Closing remarks 
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